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ABSTRACT 

This System and Procedures Exchange Center (SPEC) 
flyer/kit is based on two research efforts: a 1986 SPEC survey on the 
collection and use of management statistics related to reference 
services, and a 1987 supplemental survey on qualitative measures of 
reference services. A brief introduction describes the survey and 
discusses the evaluation of general reference services, reference 
desk service, online database search services, and bibliographic 
instruction. Trends and issues in reference service evaluation are 
also briefly discussed. The major part of this kit is a collection of 
relevant materials submitted by respondents to the survey: (1) 
policies and standards for overall reference srervices (University of 
Michigan, New York University, and George Washington University); (2) 
evaluation forms and survey results for reference desk service 
evaluation (University of Arizona and the University of Georgia); (3) 
evaluation procedures, forms, and reports from online database search 
services (University of Georgia and the University of Michigan); and 
(4) procedures, questionnaires, and survey results for bibliographic 
instruction (University of Georgia, Pennsylvania State University, 
Indiana University, and the University of California at Santa 
Barbara). A selected reading list is also provided. (CGD) 
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PERFORMANCE EVALUATION IN REFERENCE SERVICES 



Reference services are the -nost visible function of a library, 
with reference staff serving as liaisons between user needs 
and the resources of the library. The quality of reference 
services rests in the capability of staff to define and meet user 
needs. Currently, libraries are developing and using a pro- 
liferation of instruments and methodologies to evaluate 
reference services, with each library tailoring its data 
collection to its own environment and needs. 

Evaluating the perfonnance of staff in reference services is a 
useful and necessary management tool. The results from 
evaluations can be used to help staff improve performance 
and formulate personal goals, allocate staff and funds more 
effectively, and identify areas of reference services thai need 
strengthening. Evaluation also demonstrates a willingness to 
be responsive to the needs and concerns of the public. 

This flyer and kit is based on two research efforts. The first 
was a Fall 1986 SPEC survey on the collection and use of 
management statistics. (Results of this survey are available in 
SPEC Kit #134, "Planning for Management Statistics," May 
1987). An overwhelming majority of respondents indicated 
that data were collected r^ularly for those activities tra- 
ditionally considered reference services. Among the 91 
responding institutions, 87.9% regularly collected datj^ about 
general reference activities, 83.5% regularly collected data 
about library or bibliographic instruction classes, and 90.1% 
regularly collected data about computer literature searching. 
To supplement the 1986 survey, in Spring-Summer 1987. 
the researcher requested further information r^arding 
qualitative measures of reference services from 25 libraries 
reporting high activities and interest in the area. 

OVERALL REFERENCE SERVICES. Providing quality 
reference services has always been a concern in libraries. 
However, the concern has intensified over the past few years 
as more and more studies claim that only 50% of reference 
queries are correctly answered by staff at reference desks. 
Concern about staff perfonnance, coupled with a rapidly 
advancing technology, has given an immediacy to per- 
formance evaluation in reference services. WiUi the advent 
of CD-ROM and computer networks that transcend ihe 
physical limitations previously experienced in libraries, 
information opportunities are expanding exponentially. 
Clear-cut solutions to information requests are not as readily 
available amidst this menu of information resources. 



Libraries are realizing the value of long-term commitment to 
ongoing evaluation of services, as opposed to reliance on 
one-shot surveys. An effort is being made to pinpoint exactly 
what contributes to excellent reference services. Some of the 
institutions contacted have aeated proposals or policies for 
measuring the quality of their reference services. Often these 
suggested procedures draw on the criteria used to evaluate 
individual staff members during annual perfonnance reviews. 
These criteria have been broadened to create an overall view 
of reference services staff and the services they provide. 

Personal criteria (including behavioral characteristics and 
knowledge of research strat^es and information resources) 
and guidelines for services (such as service to various cli- 
entele groups) are being combined to present an overview of 
staff and services. A number of institutions have expressly 
stated that quality service is a function of quality staff: It 
often is impossible to separate personal qualities from the 
services provided. This is an undeniable given of public 
services, and for this reason, individual performance is often 
inseparable from the overall service when the evaluation 
process takes place. 

REFERENCE DESK SERVICE. Measuring perfonnance 
effectiveness at the reference desk is the most controversial 
aspect of performance evaluation in reference services. The 
literature details a number of approaches that have bee** 
used, including unobtrusive or intrusive observation, and 
evaluation by peers, users of the services, or trained proxies. 
Few institutions arj eager to initiate any ongoing, systematic 
evaluation of staff at the reference desk. In addition to the 
controversy generated by collecting subjective judgments 
about staff, another concern voiced by the institutions 
contacted included the drain on staff and finances that 
ongoing, systematic evaluation of desk service entails. 

Quantitative data collection at the reference desk is a simpler 
and more widely spread practice. Measurement techniques 
focus on tangible variables that can be transliterated easily 
into meaningful numeric data. For example, a typical mea- 
sure iient technique consists of the tallying of number and 
types of reference transactions that transpire at a service 
point during a specified time period. This type of data 
gathering will give a fairiy accurate reading of the quantity of 
service being provided, but will tell administrators very little 
about the quality of the service; in particular, it will provide 
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no assessment at ril of the behavioral characteristics of in- 
dividual staff members, or their knowlecfee of research 
strat^ies and information resources. 

Most of the institutions which conduct some sort of quali- 
tative examination of reference desk staff do so by solidting 
evaluations from either peers or users of the service. Ideally, 
both groups should evaluate desk service. Studies ha^e 
shown that user satisfaction is oftra heavily influenced by 
interaction with the staff; if the staff member has been 
pleasant, the user will evince satisfaction whether or not their 
information needs have been fully met Evaluation by peers, 
on the other hand» may lack an accurate assessment of the 
subtle nuances present in an interchange between client and 
staff. The most comprehensive view of an liltrary's reference 
desk service is presented when the quantitative data so com- 
monly collected are supplemented by (qualitative evaluations 
gathered from a variety of user groups and peers. 

ONLINE DATABASE SEARCH SERVICES. Although 
the product of online database search services an online 
database search lends itself readily to quantitative analysis, it 
is difficult to gauge the quality of the search or the searching 
techniques employed As in reference desk service, most 
institutions cumulate quantitative statistics about their 
database search services. When qualitative evaluations are 
sought, they ai'e of two types: 1) evaluation by the user of 
the search results as well as of the user's interaction with the 
staff member, and 2) a peer review process that uses criteria 
emphasizing search strati^es. Again, there is the conflict of 
user versus peer perceptions of the service. Whereas the user 
may be happy with any dtations related to the research 
topic, a colleague may detect flaws in the search strata that 
limited the number of useful dtations retrieved Evaluations 
by both users and peers, used in conjunction with each other, 
can provide valuable information about database search 
services, and in particular whether effective use is being 
made of two expensive resources— staff and electronically 
retrieved information. 

INSTRUCTIONAL SERVICES. Instructional services can 
take any number of forms. They can consist of one-time 
lectures, quarter-length classes, tours of and orientations to 
fadlities and collections, term paper clinics, and seminars. 
They can be offered to students, faculty and library staff, 
and, on occasion, to clientele outside the academic com- 
munity. Most of the evaluation of instructional services 
focuses on formal presentations; e.g.. classes or seminars 
offered to students and faculty. Qualitative evaluation re- 
garding staff performance of instructional activities is easier 
for libraries to implement than qualitative evaluation of 
reference desk and online database search staff in that aca- 
demic models for instructional evaluation already exist. 
Evaluations are sought most commonly from class or sem- 
inar partidpanls, and less commonly from library colleagues. 
Data are collected regarding the usefulness of the class or pre- 
sentation, and the teaching/interpersonal skills of the li- 
brarian. Instructional services are the only activities within 



reference :«rvices where qualitative data are collected as fre- 
quently as quantitative data. Library administrators have 
greater access to feedback regarding staff performance in the 
instructional seiung. 

TRENDS AND ISSUES. At present, qualitative data 
collection for the purpose of evaluating reference services is 
an imperfect administrative tool. Library administrators 
traditionally have relied on quantitative data for decision- 
making in reference services. Although appropriate for some 
planning processes, this type of data provides only a partial 
view of overall reference activities. 

Despite the difficulty of the task, there is increasing interest 
in measuring the effectiveness of reference services as li- 
brarians become aware of the potential benefits. As pointed 
out by Cronin ("Performance Measurement for Public Ser- 
vices in Academic and Research Libraries," OMS Occa- 
sional Paper #9, February 1985), performance measurement 
can provide administrators with an objective indicator of the 
quality of service usefid for supporting budget requests and 
planning and allocating resources. At the departmental level, 
the measures can help in setting priorities and pinpointing 
activities which require more staff^ support and training. For 
individual librarians, performance measurement provides a 
baseline for developing personal objectives. And overall, 
performance measures can encourage higher standards 
among all library staff, resulting in better service for the 
h^brary user. 

While the importance of performance evaluation is recog- 
nized, the costs in terms of staff time and funding have been 
dted as deterrents to the piu^uit of any ongoing, systematic 
evaluation. In spite of this, there is evidence of activity and 
accomplishments. The primary focus for performance evalu- 
ation in reierence services has been, until recently, on desk 
service. Currently, however, libraries are beginning to look at 
how to evaluate their entire array of reference services. Prog- 
ress also is being made toward standardization of data col- 
lection methodologies and instruments as more groups with- 
in the profession examine this issue. 

The SPEC Kit on Performance Evaluation in Reference 
Services (#139, November-December 1987) contains pol- 
ides and standards for overall reference services from 3 
libraries; evaluation forms, survey results for reference desk 
service from 2 libraries; evaluation procedures, forms, reports 
for online database search services from 2 libraries; pro- 
cedures, forms, survey results for instructional services from 
4 libraries: and a selected reading list. 

* * * 

This flyer and kit was prepared by Kathleen Gorman, Assis- 
tant to the University Librarian, University of Minnesota Li- 
braries, as part of the OMS Collaborative Research Writing 
Program. 
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INTRODUCTION 



Reference service at the Alfred Taubman Medical Library has two primary 
purposes : 

-To locate accurate and appropriate information for users within 
their time requirements, 

-To educate users in efficient methods of conducting library research 
and effective use of library resources • 

As a step in improving the quality and consistency of reference desk 
service and to increase the objectivity of performance evaluation, the 
Reference Department of the Alfred Taubman Medical Library developed a 
set of standards and identified criteria by which they could be measured. 

In order to accomplish this task, the department, working as a group, 
followed a series of steps: 

1. A preliminary list of criteria or qualities associated with 
good reference service was created. 

2. Theee criteria were reviewed and grouped into three clusters: 
behavioral characteristics; knowledge; and problem-solving skills. 

3. A literature search was conducted, relevant articles and books 
were reviewed by individuals, and findings shared. 

4. The criteria originally proposed were re-examined and a second 
list was generated. 

5. From the revised list a core of five service objectives resulted. 
These became the standards: 

a. Reference desk staff convey an attitude and manner that 
encourages users to seek assistance. 

b. Assistance is provided at the appropriate level of need. 

c. Reference librarians have a thorough knowledge of resources 
and collections. 

d. Reference librarians are able to plan and execute effective 
search strategies for complex or extended reference questions, 

e. Library services and policies are understood and described 
to users whenever appropriate. 

6. For each standard, indicators were identified - expected behavior 
or knowledge that would indicate fulfillment of the standard and 
could be used to measure performance. 
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7. To supplement tnc standard relating to knowledge of services and 
pclicies, reference services were described, with levels of service 
and policies outlined for each. 

8. In order to incorporate the standards into the annual process of 
individual performance appraisal, a checklist of reference skills 
was designed, addressing each of the standards and giving a choice 
of performance levels. 

9. A process of peer review, using the checklist, was agreed upon. 

10. A program of staff development was proposed to provide initial 
training and to stimulate continued growth and improvement of 
reference skills. 



REFERENCE SERVICE STANDARDS AND INDICATORS 

A. Reference desk staff convey an attitude and manner that encourages 
users to seek assistance. 

Indicators ; 

Acknowledgement of people who approach the desk or are waiting 
for service (when staff are occupied with another user). 

People who seem to need help are offered assistance. 

All questions are responded to in a non~ judgmental manner. 

Unusual requests for service are responded to in a positive manner. 
Possibilities are investigated or the request is referred to a 
staff member better qualified to answer the question. 

Alternatives are suggested if the request cannot be meu by the 
library. 

Time estimates for extended work are given, so requesters have 
realistic expectations. 

B. Assistance is provided at the appropriate level of need. 
Indicators : 

Ability to determine the real question; continued questioning 
to be sure the problem is understood. 

Being certain the person knows how to use the sources or she is 
being referred to; providing instruction in the use of sources if 
needed. Sources include the card catalog, union list, indexes, 
and other reference tools. 

Alternative sources are suggested, including other libraries or 
ncn-library sources. 

Services are suggested when appropriate, and pertinent information 
about them is offered, even if not directly requested (e.g. ILL, 
searches, book recommendations). 

Questions are answered within time requirments of the user. 




Reference librarians have a thorough knowledge of resources 
and ci>llections. 



Indicators; 

Ability to answer questions accurately and efficiently from 
sources in the reference collection* 

Ability Co provide accurate information efficiently about 
collection holdings and locations in the Taubman Library, 
on order, or available elsewhere on campus* 

Familiarity with and ability to apply basic catalog filing rules. 

Familiarity with changes over the years in cataloging policies and 
practices that affect holdings information. 

Ability to explain scope and comprehensiveness of Taubman collection 
and general current selection policies of other collections on 
campus. Referrals are made when appropriate. 

Ability to describe coverage (subject scope, time period, sources 
included) of relevant databases. Suggestions are made when appropriate 

Ability to explain different classifications and collections in 
Taubman. 

Reference librarians ace able to plan and execute effective search 
strategies for complex or extended reference questions. 

Indicators ; 

Potential approaches are identified. 

The alternative most likely to produce the answer in the least 
amount of time is selected. 

The most complete answer is found with use of a minimum of resources. 

The questions are answered accurately, appropriately, and within time 
requirements of the requester. 

Technical proficiency in NLM and BRS system searching is maintained. 

Library services and policies are understood and described to users 
whenever appropriate. 

1. Literature searches 
Indicators : 

Advises on appropriateness of computer search on alternative 
data bases. 

Provides information on alternative data bases. 

Informs new requesters of fee schedules, major search options 
(time periods, abstracts, online retrieval), turn-around time. 

Suggests SDI searches. 

15 
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Able to distinguish use of reference search » quick search^ and 
regular search options and to apply them appropriately. 

2* Material not owned 
Indicators ; 

Determines if on order or in-process. 
Identifies other campus locations. 
Offers notification service. 

Encourages requester to make purchase recommendations. 
Suggests interlibrary loan. 

3. Interlibrary Loan 
Indicators : 

Able to describe copyright compliance laws as they apply to 
limits on photocopy requests. 

Verifies that item is not owned on campus. 

Determines accuracy of citation if necessary. 

Informs requester of estimated turn-around time» 

Accepts requests only from eligible users; suggests 
alternatives for others. 

4* Intralibrary loan and storage 
Indicators : 

Familiar with libraries from which loans are possible. 
Accepts requests from eligible users only. 
Checks forms for completeness and accuracy. 
Knows what materials are in storage. 

Informs users of turn-around time and pick-up location. 

5. Photocopy services 
Indicators : 

Can describe different photocopy services available. 
Knows charges and billing requirements. 

6. Loan services 
Indicators : 

Can dsscribe normal loan periods for material types. 

Able to locate and interpret circulation records in Geac- 
both in Public Query and staff functions. 

Can interpret policy when necessary (if Circulation 
supervisory staff is not on duty). 

7. Tours and instructional programs 
Indicators : 

Able to explain clearly the layout and arrangement of 
collections . 

Can describe programs available on request for orientation 
and instruction. 



^J^(^ Suggests orientation programs when appropriate < 
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8* Rare book collection access 
Indicator s : 

Able to identify Rare Book Room materials* 

Locates materials in Rare Book Room. 

Describes policies for access and use of Rare Book 
Room materials. 

Can describe policies and procedures for duplicating 
Rare Book Room materials ♦ 

9. Department library consulting 
Indicators : 

Able to identify standard reference materials, textbooks, 
and journals appropriate for department library in health- 
related disciplines. 

Able to recommend organization and procedural alternatives 
for department libraries appropriate to local collection 
and needs. 



III. SERVICE POLICIES AND GUIDELINES 



A. User groups served by the Alfred Taubman Medical Library 

1. Primary clientele: 

Faculty, students, and staff of the Medical School, the 
School of Nursing, the College of Pharmacy, and University 
Hospitals. 

2. Secondary clientele: 

Faculty, students, and staff from other divisions of the 
University of Michigan, 

3. Tertiary clientele: 

Health professionals working and/or residing in the 
immediate locale. 
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4. Quartemary clientele: 
The general public. 

Levels of service provided to each user group 

1. Primary clientele: 

Primary clients are eligible to borrow TMl materials, to 
request computeri^^ed literature searches on a fee-for 
service basis, to request materials not owned by the 
library on interlibrary loan, to obtain extended reference 
assistance in finding information or solving complex and 
extensive bibliographic problems, to use all photocopying 
services, and to arrange for specialized instructional 
programs on the use of TML resources and services. 

2. Secondary clientele: 

Secondary clients have access to the same privileges outlined 
above for primary clientele, with some limitations on the 
instructional programs offered. Those programs will be 
available to secondary clients when it is evident that the 
nature of their work requires instruction in the use of the 
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health science literature. Inter] ibrary loan materials of a 
health science nature will bo reouested for secondary clientele. 

3. Tertiary clientele: 

Tertiary clients may borrow T>fL materials, utilize the services 
of the reference staff, request computerized literature 
searches on a fee-for-service basis, and request interlibrary 
loans for materials not owned by the library. Formal instruc- 
tional programs will not be held for non-U. M. users. 

4. Quarternary clientele: 

Quarternary clients may utilize the facilities of TML, and 
obtain assistance from the reference staff in finding infor- 
mation, solving complex and extensive bibliographic problems, 
and other services as deemed appropriate by the reference 
librarian. Requests for computerized literature searches will 
be referred to Michigan Information Transfer Souce (HITS), 
Since the library does not collect materials for the general 
public, every effort will be made to refer laymen to appropriate 
sources and collections of information. 

C. Guidelines for specific services 

!• Telephone requests for reference assistance 

Both in-person and telephone requests for reference 
assistance are accepted. 

On-site requests normally take precedence over telephone 
requests, but all requests should be answered as promptly 
as possible. 

The number of requests for verification of citations, 
holdings information, or other similar questions accepted 
by telephone may depend on circumstance. The librarian 
should make a reasonable judgement, and offer a positive 
alternative beyond what can be done immediately. 

When on-site problems necessitate the librarian being 
away from the desk, the assistance of another staff 
member should be sought, an appointment made to continue 
with the problem later, or another solution found that is 
suitable to the requester, 

2 . Online reference searches 

Online searching at the reference desk provides another tool 
for the staff to use in answering reference questions. The 
following criteria are designed to ensure consistent use of 
this tool in order that library users and staff can distinguish 
between the fee-based service and the reference search. To 
qualify as a reference search all of the following statements 
should be true: 



The question is most effectively answered using an 
online system. 

Using an online system will not delay service to 
other users . 

The request suggests a simple formulation which should 
yield a single piece of information. 
An online system will require less time to answer 
the question than consulting a printed index or other 
reference tool. 

The search can be run in less than 5 minutes. 
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Requests : 

Database searches are available to primary, secondary and 
tertiary user groups, on a fee basis. Others are referred 
to MITS. 

Although it is preferred that search requests be made in 
person, requests may be made by telephone or in writing if 
more convenient for requester. Clinical staff are en- 
couraged to request searches by telephone. 

Results : 

Searches will be run within twenty-four hours, and up to 
twenty citations (without abstracts) printed online if 
desired. A greater number of citations may be printed on- 
line for an additional fee. 

Offline prints are generally received within one week. 

If the requester is dissatisfied with the results of a 
search and the searcher believes there is potential for 
improvement, the search will be rerun at no additional 
charge . 

Quick Search: 

The purpose of the Quick Search is to provide a few 
current references quickly at a minimal cost. Searches 
will be run within 24 hours. A Quick Search is appropriate 
if : 

-Search does not exceed ten minutes online time (or 15 
citations printed), whichever comes first. 

-The current file of MEDLINE (current 4 years in BRS) 
is the only file to be searched. 

-The citation printed will consist of author, title, and 
source only. No abstracts will be printed. 

Selective Dissemination of Information (SDI) Searches: 

-Searches are run monthly and billed quarterly. 

-The minimum contract period is four months. Updates 
requested for one to three months must be requested 
individually and regular search fees apply. 

Orientation and Instruction ; 
a* Tours: 

The reference staff will provide primary, secondary, and 
tertiary clients with orientation tours of the library. 
Tours will be offered annually at the beginning of the Fall 
term. Individual or group tours will be arranged to 
accommodate special user needs or schedules. 



b . Library-Use Inst rue t ion : 

Seminars providing an in-depth look at specialized 
library research tools, resources, and services will 
be arranged to meet specific information needs of 
primary and secondary clients only. 

Librarian instructors will provide formalized instruc- 
tional programs for nursing, pharmacy, and medical 
students . 

Librarian instructors will serve as special liaisons 
to the Schhol of Nursing and the College of Pharmacy. 

One member of the department will serve as coordinator 
of user education activities. 

c. Instruction in the Use of Search Services: 

Demonstrations of computerized search services will be 
provided to primary and secondary clients only. 

Demonstrations are designed to familarize patrons with 
file capabilities and coverage, while also teaching the 
essentials of search strategy formulation. 

5. Interlibrary Loan (ILL) 

ILL requests are accepted by telephone and in-person. 

Primary, secondary, and tertiary user groups may borrow 
materials on interlibrary loan; other requests will be 
accepted if reasonable and the person has no other direct 
access. ILL requests are verified and forwarded to ^LL 
office within 24 hours. 

6. Consultation and Referral Services: 

a. Michigan Information Transfer Service (HITS) 

When the librarian on duty at the reference desk determines 
that the patron is not eligible for service at TML, the 
librarian will inform the client of the services offered 
by MITS. 

b. Locating Information Elsewhere on Campus: 

When the reference librarian determines that the TML does 
not own the item sought, it is that librarian's responsibility 
to determine if the material is available at another U.M. 
library . 

Standard bibliographic tools should be used to verify the 
accuracy of the material sought. 

Common sense should dictate the number of items to be 
handled. The performance of this service should not 
significantly slow down other reference desk work. Should 
that occur/ the librarian has the option of giving the user 
the telephone numbers of the aforementioned units and suggesting 
that he place the call himself. Additionally, the librarian 
can also offer to provide the infoirmation at a later time. 
This policy applies to both on-site and telephone requests. 
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c. Departmental library consultlnR. 



TML reference librarians will be available to provide advice 
on the establishment, organization, and maintenance of 
departmental libraries for the School of Nursing, the 
Medical School, the College of Pharmacy, and University 
Hospitals. Such requests should be referred to the Hec.d 
of the Department, who will arrange for a consultant. 

7 . Access to Rare Book Collections 

The librarian on duty at the reference desk will accept 
requests to use Rare Book Room material* He will notify 
the backup librarian whose responsibility it is to retrieve 
the needed item. 

In-depth questions regarding the use of Rare Book Room 
materials should be referred to the Rare Book Coordinator. 

IV. PERFORMANCE EVALUATION 

The standards form a basis for measuring individual performance. They 
are used in the annual performance appraisal process and to improve 
the quality of service provided. 

The review process includes the following steps: 

a. A checklist of reference skills (see attachment) provides 
a means for peer review. 

b. Two librarians are scheduled at the reference desk during 
the busiest hours, and assignments are rotated to allow 
each librarian to work with all others. 

c. Each reference librarian completes a checklist for every 
other member of the department at four^month intervals. 

d. Checklists are submitted to the department head for review 
and follow-up discussion. 

e. Information from these reviews are a component of the annual 
performance evaluation. 

V. STAFF DEVELOPMENT 

a. Reference questions requiring more than 5 minutes to answer are 
recorded on a work form. Tools used and strategy followed are 
reported on this form. The librarian is not identified. 

b. The department head periodically reviews the forms to assess general 
knowledge of reference sources, search strategy and problem-solving 
skills. 

c. Areas in which staff training or development are needed are identified 
through re* lew of questions and search strategies. 

d. Group review sessions are scheduled to share knowledge of resources, 
and problem-solving techniques. 

e. Discussions include printed reference sources and online systems 
or databases. 

f. Basic skills such as interviewing are reviewed as needed. 
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NAME 



CHECKLIST OF RKKERENCR SKILLS 

DATE 



ATTACHMENT 



To complete vt^is cno^^klis: pio.v'e assess each librarian's skills using the 
following parameters: 1) us<;all> perfcrms at this level, 2) performance needs 
improvement, 3) performance n."t observod. Select the number to the left of the 
skill that best indicates levo', of performance. If during the time period 
involved you were not able to assess this individual's performanco, plenst- 
select number 3. Place an X in the ippropriite space. 



SKILL 




LEVEL 


OF 


PERF0R>-1ANCE 


ArTIlUUE AND DEMEANOR: 










1. friendly attitude 


1. 


(1) 


(2) 


(3) 


2. acknowledges users who approach desk 
are waiting for assistance 


2. 


(1) 


(2) 


(3) 


3. assistance is offered to those in need 


3. 


(1) 


(2) 


(3) 


4. responses are non-judgemental 


4. 


(1) 


(2) 


(3) 


5. responds positively to unusual questions 


5. 


(1) 


(2) 


(3) 


6. stiggests alternatives when Taubman Medical 
Liibrary (TML) cannot service request 


6. 


(1) 


(2) 


(3) 


INTERVIEWING, LISTENING, REFERRING: 










1, skillfully interviews users 


1. 


(1) 


(2) 


(3) 


2. determines patron's ability to use source 
referred to 


2. 


(1) 


(2) 


(3) 


3. instruction provided when appropriate 


3. 


(1) 


(2) 


(3) 


A. sources recommended are ar appropriate level 


4. 


(1) 


(2) 


(3) 


5. regularly suggests alternate sources 
or services 


c 

5 . 


(1) 


(2) 


(3) 


SEARCH STRATEGY: 










1, identifies alternate approaches 


1. 


(1) 


(2) 


(3) 


2. chosen strategy is well planned 


2. 


(1) 


(2) 


(3) 


3. finds complete answer with use of minimum 
of resources 


3. 


(1) 


(2) 


(3) 


^. answers are appropriate, accurate and meet 
user's time frame 


4. 


(1) 


(2) 


(3) 


KNOWLEDGE OF RESOURCES AND COLLECTIONS: 










1, answers questions accurately and efficiently 
using reference collection 


1. 


(1) 


(2) 


(3) 



ERIC 
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ATlACaiLN 



SKI I.!. 



LE\F.L OK PERFOR>IANCE 



3. 
5. 



determines colJoction holdiiiRS information 
•ind/or status for TML and lliiivcrsitv Library 
(UL) by usinf>: 

card catalog 

h. RLIN 

c. Union List 

d. 1EAC 

explains scope and selection policies of TML 
and UL collections 

describes coverage (subject scope, time period, 
sources included) of relevant databases 

effectively uses online sources when 
appropriate 



2a. (1)_ 

b. (1)_ 

c. (1)_ 

d. (1) 



3, 



(1) 



-(2). 
-(2). 

(2) 



-(3). 
(3) 



.(2) (3)_ 

.(2) (3)_ 

(2) (3) 



TML SERVICES AND POLICIES: 

1. identifies users eligible for service 

2. knows level of service provided to each 
user group 



1. (1) (2) 



knowledgeable of full range of services provided 
at reference desk 3 

knowledgeable of full range of services provided 
by Reference/Information Services department, TML 
and UL, including: 

online search services 

user education 

collection development 

interlibrary loan 

intralibrary loan 

photocopying 

consulting on departmental libraries 
MITS 



a. 
b. 
c . 
d. 
e. 
f. 

8- 
h. 



Ua. 
b, 
c . 
d. 
e . 
f . 

g. 
h. 



(1) (2) 



(1) (2) 



H). 
(1). 
(1) 



.f2). 
.(2). 
.(2). 

.(2)_ 
.(2)_ 
.(2)_ 
(2) 



(3) 



(3) 



(3) 



.v'3). 
.(3). 
.(3)_ 
(3) 



Comments ; 
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NEW YORK UNIVERSITY 



FIVE YEAR PU^N 
TASK PC^CE REPORT ON THE EVALUATION OP REFEREIKE SERVICE 



Task Force Members: Mary Henry (Chair), 
Angela Carreno, Maryann Chach/ Ree DeDonatO/ 
Richard Glasford, Ann Volpe 



Reference centers at Bobst Library strive to maintain a high 
standard of service in responding to the information needs of 
students and faculty of NYU and affiliated institutions. This has 
become a greater challenge in recent years with the growing 
complexity of libraries and library technology, the proliferation of 
services provided by reference librarians/ and the increasing 
sophistication of library users' information needs. These major 
changes in technology, services and resources are transforming the 
reference librarian's work. Reference must now be viewed within the 
larger context of the growth of technology, and traditional views 
based on past experience and practice nust be revised to include new 
possiblities for reference in the future. 

Bobst Library has the central role in the delivery of reference 
services at New York University, and in the years ahead/ it must 
continually evaluate its effectiveness in this role. Future 
performance measurements must be compatible with the library's human 
and financial resources. They must also be specific to the purpose 
and goals of each reference center. Individual performance appraisal 
is an important component of the overall evaluation of reference 
service. In addition to ongoing evaluation of prograns and 
personnel, continued planning for reference is also essential and 
should be integrated with overall planning for public services. 

Key issues in evaluating reference service and planning for its 
imprwement in the years ahead are described below. 



1. Reference Training, Staffing and Staff Developme nt 

The reference librarian's role is a multifaceted one which typically 
includes reference service on and off the reference desk, 
bibliographic instruction, online searching, and collection 
development. In recruiting new reference faculty, the library seeks 
candidates with strong backgrounds or potential in both public 
services and collection development. This blend of activities is 
appropriate for reference librarians, since public services and 
collection development are complementary activities. 
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Bob6t Library Pivie Year Plan 
Reference Servrice 
Page 2 



Recruiting of new faculty with diverse skills means that public 
services orientation must be tailored to each librarian's needs and 
each reference center's demands. Orientation of faculty is not 
currently emphasized/ and this may need to be reassessed. New staff 
training techniques ai^ aidS/ such as detailed procedural manuals 
identifying areas of responsibility and goals for individual services 
for each reference center/ should be considered. The training of 
reference support staff is important and also requires due 
consideration. 

Staff development activities have been actively supported at Bobst 
Library and have enabled reference librarians to develop skills and 
learn about other centers and departments. In the years ahead/ it 
will be necessary to provide even more opportunities to learn about 
new technology/ new developments in technical services/ and library 
and camj^as resources/ activities and policies. Unexplored staff 
developcrent opportunities are numerous. Two that are strongly 
irecomroended are '^iveloping and maintaining professional contacts with 
other institution in the city/ and increasing familiarity with city 
resources « 

Recent job exchange arrangements have acquainted participants with 
different clientele/ reference resources and services in Bobst 
Library and elsewhere on campus. Participation should be broadened 
to develop a base of comron experience and knowledge/ to improve 
referral sen/ice/ and to increase the pool of qualified reference 
staff available during staff shortages. A mechanism for exchanging 
infonnation and experience with technical services librarians must 
also be considered. 

Reference staffing/ a recurring problem/ can be addressed by defining 
and maintainincf adequate professional and support staffing levels to 
insure optimum quantity and quality of reference service. Using 
reference statistics currently collected can also assist in 
evaluating and implementing staffing patterns which would avoid 
disruption of other public service and collection development 
activities for which reference staff is responsible. To maintain the 
required staffing levels/ contingency plans should be developed to 
provide for adequate staffing during professional leaves/ vacancies 
and recruitment. These could include "hotline" advertising, hiring of 
temporary librarians/ recruiting and training of staff fron oth3r 
departments, and compiling and drawing from lists of volunteers on 
call. 
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Bobflt Library Five Year Plan 
Reference Service 
Page 3 



2. Automation's Impact on Reference Service 

Access to new technology, principally BobCat and RLIN to date, has 
improved the qualit> of reference service by enabling reference 
librarians to respond more quickly and effectively to patrons' 
information needs. As new technology evolves, reference librarians 
mL^st keep pace with the changes and develop and refine their 
searching skills in current and future databases. The demands on 
reference librarians are becoming more complex in response to the 
growing complexity of the library and research environment. Remote 
access to the library's catalogs/ direct patron access to commercial 
and specialized databases aiid the use of personed corrputers are three 
areas wl-iich are already affecting the role of reference librarians. 

BobCat user studies thus far have provided valuable information on 
current patterns of online catalog use. Data on user expectations 
and user performance which is necessary in planning for BobCat 
instruction needs to be gathered on a continual basis. Emphasis 
muse now shift from user satisfaction to more efficient user 
performance as the BobCat database grows and users become more 
familiar with contputer technology, BobCat instruction at service 
desks has provided reference librarians many more opportunities for 
contact with patrons and for learning about users' approaches to 
information technology* This experience should be drawn upon in 
improving BobCat use instruction and in planning end-user searching 
instruction in the years aheari. 

Familiarity with user approaches to library research has et-iabled 
reference staff to play an active advisory role in the development 
of BobCat. Future BobCat enhancements recommended are: 
center-specific sorts; subject authority online; author authority 
inprovement; inproved browsing capability; a stop-list for foreign 
languages; simplification of uniform-title searching; and searching 
capability for new titles, by subject. Also recommended is the 
appointment of a standing committee of public ser^/ices and technical 
services personnel to evsduate and make recommendations concerning 
BobCat/ RLIN and technology yet to be implemented. 

RLIN provides acquisitions information not previously available as 
well as faister access to current information on holdings of local and 
regional libraries to which patrons are often referred. RLIN's 
special files are increasing the awareness of and demand for 
specialized' research materials the library mir[ht provide. Although 
there is currently no printing or direct patron access for RLIN at 
NYU, the public need and the capability to provide it are a reality. 
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?age 4 



3. Reference Facilities 



As the scholarly environment is transfonned by new technology, the 
delivery of traditional reference desk service is also being changed 
Online access at NYU to campus, regional and national resources makes 
reconfiguration of reference service likelv. Planning for this 
reconfiguration is critical. Reference librarians will have to 
evaluate the impact of new technologies on reference service anfd 
advise on the new directions that reference service should tak=' 
Whether ana how much change can take place will depend on its 
anticipated impact on the quality of service, and on staffing and 
equipnient issues. The flexibility that new technology affords 
liorarians may result in more personalized reference service and 
better utilization of specialists' skills. 

To benefit fully from new technology, reference librarians must have 
broader access to BobCat, RLIN and personal computer terminals and 
printers in their service centers and office areas. Personal 
computers, scon to be available to reference librarians, will improve 
service by facilitating access to other forms of resources and 
infonnation not easily consulted in their present fonnat, such as 
individually created bibliographies and records. Personal corrputers 
and printers will also siirplify the development and updating of 
instructional materials. 

Existing space nust be used to best advantage, and space eivailable 
for future reference facilities mst be assessed. Planning for 
growth and the use of reference materials in new formats also needs 
to be considered. 



4. Reference Collection Management 

In-depth assessment of the resources of each reference center is 
needed to determine their adequacy in serving the needs of reference 

patrons. The reference collection development policy 
should be revised as needed to provide selection guidelines for these 
materials, and to reflect changes in selection policy as they occur. 
Coverage of current and retrospective reference sources in unassigned 
subject areas needs to be improved. 

Reference materials in new formats, such as data files, are expected 
to proliferate in the years to cane, and reference librarians must be 
able to evaluate and select materials appropriate for use in the 
reference centers. 
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5. Bibliographic Instruction 

A wide variety of traditional and new instructional avenues are 
available to facilitate better independent exploitation of the 
library's resources, including new rt^edia. In the years ahead, 
nx- luers of the reference staff should advise the new Reference 
Librarian for Instructional Services on improving instructional 
methods and materials. This important component of reference service 
is discussed in greater detail in the bibliographic instruction 
section of the five-year plan for public services. 



6. Referrals 

To insure consistent delivery of service, a clarification of 
reference policy with regard to referrals within Bobst Library is 
needed. Standardized referral procedures would clarify how much 
groundwork should precede referrals to other reference ' centers and 
9ther libraries, and when telephone verificatl -^n is necessary. How to 
better familiarize staff with resources within the library, the 
university and the metropolitan area should also addressed. 
Procedures and policies for public service referrals to technical 
services units should also be evaluated to improve efficiency and 
ease of access to library information. 



7. Other Public Services Areas 

Reference-related assistance is routinely pzrovided at the following 
service points: Reserve, Current Periodicals and Microforms on the A 
level, the Avery Fisher Center for Music and Media, Pales Library, 
Tamiroent and University Archives. How to improve communications and 
increase familiarity between these areas and the reference centers 
needs to be addressed. Selected basic reference sources should be 
acquired for study areas with extended hours, and the Collection 
Development Ccmmittee should determine the content , size and location 
of these resources. 

Recurring problems in communications and materials flow between the 
reference departments and Circulation and Stacks must be addressed in 
order to improve reference service in the years ahead. Problems 
interfering most with reference service need to be resolved so that 
reference staff can perform more effectively, stacks management is 
the most irtportant of these issues, and one which affects reference 
most directly. Other issues include change and photocopy machine 
problems, room assignment issues, and inadequate and misleading 
signage. 
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Reference & Instruction Department 
July 24, 1987 



Dear Colleague: 

I enclose copies of the perfonnance standards we have defined 
at Gelman Library for the areas of reference service, bibliographic 
instruction, and online searching which you requested as a result 
of discussions at the ALA Conference in San Francisco. While these 
standards are specific to the context of reference librarian 
positions at Gelman, we found some of the previous literature in 
this area to be very helpful in preparing for our departmental 
discussions, specifically: 

1) Adams, Mignon S. and Blanche Judd, "Evaluating Reference 
Librarians: Using Goal Analysis as a First Step," The 
Reference Librarian 11: 131-145 (Fall/Winter 1984). 

2) Schwartz, Diane G. and Dottie Eakin, "Reference Service 
Standards, Perfonnance Criteria, and Evaluation," The Journal 
of Academic Librarianship 12: 4-8 (March 1986). 

3) "CRD Reference Desk Perfonnance Standards (Evaluation Form)" 
and "CRD Reference Desk Perfonnance Standards/Peer Review 
Form," from the University of Arizona Library, unpublished 
but supplied on request from the Uni«'. of Arizona Library. 

We found fewer direct models in the literature for bibliographic 
instruction and online searching, although there were a number of 
checklists and documents we examined that dealt with evaluation 
of teaching and public speaking and considered for bibliographic 
instruction, and there was a checklist of "Points to Consider in 
Evaluating the Effectiveness of Search Analysts" developed by 
the Computer Search Sen'ice at the State University of New York ' 
at Albany Library that was helpful in the online searching area. 

Some of the items in the enclosed standards reflect the 
organizational structure and internal communication mechanisms 
in our particular institution, e.g.. Renaissance Times is the 
weekly library internal newsletter, and all librarians at Gelman 
Library, both within our department and in otner library departments, 
serve as subject specialists with collection development and faculty 
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responsibilities. If you have questions about what we mean in 
any of the performance standards, feel free to contact me for 
further explanations. 

The standards themselves were developed by the department 
at an all-day retreat this spring. In preparation for the 
retreat, the departmental coordinators for reference service, 
bibliographic instruction and online searching conducted 
literature searches in their respective areas, distributed 
copies of the most relevant materials to all department 
members, asked for individual responses identifying the 
broad criteria and suggested performance standards within 
them for each area, and distributed compilations of those 
individual responses to all department members prior to 
the retreat day. The day itself was spent in group process 
work discussing the compiled responses and reaching consensus 
on retaining, revising or eliminating items from those lists. 

We have not yet addressed the issue of how to use these 
standards, although we see them as providing checklists of 
specific behaviors that individuals can use in preparing self- 
evaluations as a part of our performance appraisal process, 
and also as a training tool for new staff. Our Coordinator 
for Bibliograpihic Instruction has also used the performance 
standards in that area as a basis for designing an evaluation 
instrument for BI sessions to be distributed to faculty and 
students who participate in those sessions. Perhaps at a 
future ALA meeting, I will have an update on how we have 
actually put these standards to use! 

Tm pleased at the interest expressed in our work on 
these standards and would certainly welcome any comments 
you may have after reviewing them. 



Sincerely yours, 

Deborah C. Masters 
Head, Reference & Instruction 
Department 
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THE 

GEORGE 

WASHINGTON 

UNIVERSITY 



neGtlmm Library/ 21 30 H Street, N.W. / Washington, D.C 20052/(202) 994-6455 



REFEP.EHCE CEP.VICK PEBFOPriANCE ilTANDAHDS 



Gonl: To provide quality roforcncr" service through: 



1. 



Knowledge of collections, po] icio 



pnd r 



csourccs? 



2. 



A v/orking rapport with pntrons; 



Cooperation with colleagues. 



Criterion 1: Knov/lodqr cf colloctjonn 



poli c:ies. 



nncl 



resources. 



Porformanco Gtandards: 

1. Continuo to develop hnov/l-dgo of collections, policies, and 
resources through such activities as: 

— Exrimininq new refcrcnco tools 

— Att<-nding colleagues' sub ject^-relcitod bibliographic 
instruction dosses 

— Participating in (attending or presenting) v/orkshops on 
reference tools and service (interna.] and external) 

— Pvcads Renaissance Times and the reference clipboard. 

2. Uses ready reference materials (e.g., Sheehy and Daniells 
guides to sources). 

3. Consults appropriate r^f prince scurcps in sensible order. 

4. Uses desk filrs (Rolod'^x, ^.nnual Report fi]e. Bindery file, 
etc . ) . 

5. Uses department's bib] iographies . 

6. Asks questions and seeks help frcn colleagues after 
exhausting obvious sources. 

7. Refers in-depth questions to subject specialist. 

8. Uses online services for ready rpfercnce v/hen appropriate. 

9. Recommends materials and strategies consistent with the 
user's level of expertise and particular needs. 

10. Informs patron^": of service such as Rcsearci Consultations, 
Term Paper Assistance, and Online r?earching, v;her^ appropriate. 

11. Explains use of sources to users. 

12. Appropriately refers users to other departments, service 
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Reference Service 2 



desks, collections, libraries, i ndivi (Tunis, etc. 



Criterion 2: A working rapport v/ith patrons • 
Performance standards: 

A. Attitude and Demeanor: 

1. Encourages initial interaccion through such activities as: 
— acknowledging users v;ho arc waiting for help 

— not appearing preoccupied 

— appearing approachable by smiling, establishing eye 
contact, etc. 

2. Maintains courtesy, patience, and tact in dealing with 
patrons. 

3. Reassures unskilled, unsure users. 

4. Louvos desk to assist user, if appropriate. 

5. Listens attentively to questions. 

6. Speaks clearly; volume and tone of voice appropriate. 

7. Balances demands during periods of peak activity. 

8. Remains composed during periods of peak activity. 

9. Shows a v/illingness to go beyond the minimum for primary 
clientele . 

B. Interviewing 

1. Structures reference intcrvie\/s to elicit information about 
nature of requesL and patron status, and formulates an 
appropriate response. For example, offers multiple or 
alternative approaches to patron questions; provides effective 
explanation of outside resources offered when appropriate. 

2. Varies communication to suit patron background. 

3. Explains library terrninology . 

4. Encourages patrons to return i f they are unsuccessful or nee 
more help. 

note; Issue of dealing v/ith difficult patrons identified as 
needing more departmental discussion. 
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Criterion 3: Cooperation with Colleagues 

Performance standards: 

1# On time for desk assignments. 

2. Constructive response to environmental changes such as 
changes of location, system up or down, disappearance of sources 
from behind the desk, demise of tho card catalog, extremes of 
temperature. 

3. Returns secured reference sources to truck before leaving 
desk. 

4. Identifies and responds to colleagues* need for assistance. 
For example, helps out when not assigned at the desk; looks back 
occasionally to check colleague loft alone at reference desk; 
explains to patron a need to return to cover reference desk if 
research question becomes over long. 

5. Offers assistance to colleagues v/hen requested. 

6. Suggests Term Paper Assistance or Research Consultation 
appointment if reference question is extensive. 

Based on Reference & Instruction Department retrent 5/9/87 dcm 
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THE 
GEORGE 
WASHINGTON 
UNIVERSITY 

The Gelmm Library 1 2 J SO H Street, N, W. / Washington, D, C 20052 / (202) 994-6455 



PERFORIIAHCE STANDARDS FOR Ol^LJUP rFARCIITnO 
Criterion 1: Kno\7]edg'^ of onJino r.yst^^mn* 
Pcrfonnanr^^^ 5t'indn?-ds : 

1. Uses Booior'in logic ^-^f iVicl.iv.^ly . 

2. Uses commands v.ru^ i -^m i fiii»^s r^pprcpri-^ to to system usvd ( 
e.g., trurc<ition syir^bols). 

Note: Appcndi:: to br ^icr/olopnd on conpvmds .^nd techniques 
relevant to eacli of systems us«^d. 

3. Choice of system f^nd search s trnf^qi es reflect n knov/lrdge of 
system limitations and pricing structuroc. 

4. Uses proper sign-on and sign-off procoduros, v/ith and without 
automatic log-ons . 

5. Keeps up-to-date on syst'-n devolopnonts by att^-^nding training 
sessions and reading update information from systems. 

Criterion 2: Kno\/]edgo of online databases. 

Performance s tandards : 

1. Uses knowledge of databns.-s to ass^^^.qs v/hether a search is 
appropriate. 

2. Constructs workable search strategies by choosing appropriate 
databases, using appropriate thesauri and free-text search 
techniques, and making appropriate use of fields. 

3. Judges v/hcn a ready reference search is appropriate and 
performs ready reference searches on the appropriate system if 
able. 

Note: Need to further define departmental policies/procedures on 
performing ready reference searches when system needed is not 
known by librarian involved in reference transaction. 

4. Serves as a resource person in subject area databases, and in 
team subject area databases. 

5. Reconmiends other reference sources which will complement 
online searching. 

6* Alters search strategy as needed (online or off). 

o4 



Online 2 



7. Keeps up-to-date on database dovolopmcnts -and now syst 
features by attending training sessions and reading update 
information from database vendors and producers. 

0. Investigates and/or seeks advice regarding unsatisfact 
search results. 



Note: Need for followup on responsibilities as subject 
specialists to alert faculty and students in subject area of 
online developments which affect them, and taking a le«-^d in 
evaluating the appropriateness of new databases/systems as they 
relate to theii subject area. Are these responsibilities within 
online searching or as subject specialists? Do they apply to 
both searchers and non-searchers, members of Reference & 
Instruction Department only or also subject specialists outside 
the department? 



Criterion 3: Effective Communication 



Performance standards: 



1. Conducts presearch interviews to assess whether a search is 
appropriate and to develop a basic grasp of the research need* 

2. Listens to user explain search question; asks questions to be 
sure search request is understood; helps users define unclear 
search topics. 

3. Puts users at ease and shows an interest in the search. 

4. Explains searching terminology. 

5. Explains in general terms how a computer search works. 

6. Tells users reasons for choosing particular databases. 

7. Explains advantages and limitations of computer searching 
when relevant. 



8. Explains search strategy: paragraph qualification, 
truncation. Boolean logic, positional operators, printing 
formats, and other special features used. 

9. Explains choice of vocabulary and use of thesauri. 

10. Shows user sample record and explains paragraph labels. 

11. Explains costs before search. 

12. Involves user in the search process and the evaluation of 
results as time' permi ts. 
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Online 3 



13. Providos follov/- through nssistnnco to lielp tho patron 
understand the results, locntc tho mnterials cited, and to 
determine the next stops. 

14. Is courteous and patient during tho search appointment. 

15. Instructs student/f nculty/stnf f groups in DRS/Aftor Dark 
self-searching and consults with individuals on their search 
strategies. 

16. Prompts self--searchers at a lovol appropriate for their 
expertise . 

Criterion 4: Reference Dopartmont/Sub ject Specialist Team Player 



Performance stanc^-^rds: 

1. Responds to requests for assistance with searching on 
databases in own or team subject areas, or using system 
expertise. 

2. Accommodates reasonable requests for changes in search 
schedule . 

3. Consults with colleagues if unsure of contents of database, 
search strategy, and system features. 

4. Incorporates information about searching into bibliographic 
instruction sessions v;hore approprlato. 

5. Invites subject specialists to scheduled searclios in their 
areas, where appropriate. 

6. Shares "discoveries" with other staff members and the 
Coordinator for Online Searching. 

Criterion 5: Knov/ledge of Procedures and Equipment. 
Performance standards: 

1. When referring patron for scheduling of search appointment, 
identifies subject of search and system to be searched, and makes 
sure search request form is filled out completely. 

2. V7hen ma?:ing appointments for searches, identifies the most 
appropriate appointment slot based on appointment schedule, 
patron's schedule, and inforination on search request form. 

3. Alerts assigned searcher to any special details. 
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4. Bills patrons correctly. 

5. Records search statistics correctly and promptly. 

6. Uses searching equipment effectively. 

7. Uses SMARTcom software effectively. 

8. Uses basic troubleshooting techniques to analyze problems 
with equipment, locates the probable cause of problems, and 
corrects straightforward machine problems such as paper jams, 
loose or unplugged cables, rebooting system, ink replacement, 
etc. Reports more complex problems to appropriate' departmental 
sta£r. 

Based on Reference & Instruction Department retreat 5/9/87 dcm 
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THE 
GEORGE 
WASHINGTON 
UNIVERSITY 

The Geimm tibrmy / 2130 H Street, N. W. / Washington, D.C 20052 / (202) 994-6455 



BIBLIOGRAPHIC inSTRUCTION PERFOPMArJCE STANDARDS 
Criterion 1: Libr<'\rian is woll-pr(}po.rod for class. 
Performance standards: 

1. Discusses v/ith faculty mcrnbor nssignmont, if any, and goals 
for BI session. 

2. Familiarizes self with basic content of course and sources to 
be included in session. 

3. Prepares outline, list of sourcos, objectives for session. 

4. Prepares handouts and visual aids to facilitate learning, 
consulting teaching fil^i to see if similar hanuouts or visual 
aids are available. 

Criterion 2: Librarian presents ideas effectively during class 
session. 

Performance standards: 

1. Communir-ites goals of session to sLudents. 

2. Makes cffp^ctivr use of visual aids when appropriate. 

3. Focuses on key concepts students must learn and tries not to 
cover too much. 

4. Incorporates exercises or other learning activities into 
presentation v/here appropriate. 

5. Encourages students to seek further assistance if necessary. 

Criterion 3: During session, librarian responds to research 
needs and level of class. 

Performance standards: 

1. Solicits questions from students and participation of 
teacher. 

2. Checks student comprehension through questions, class 
activities, etc. 

3. Encouragos all students to participate. 

4. Varies planned session and improvises as necessary. 

29 
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Criterion 4: Llbrnrian's spoocli and non-vorbnl conmunj cation 
expedites learning . 

Performance standards : 

1. Speaks and acts in a confident manner. 

2. Speaks clearly, at appropriate volume, and at appropriate 
pace . 

3. Explains any necessary library t^^rminology . 

4. lias good eye-contact with students. 

5. Behaves courteously toward class. 

6. Communicates own inlM»r'^3t in concepts and skills taught. 

7. Minimizes districting mannorismn. 

Noto: I'old for further departmental djr;cussiont Classroon 
management techniques, ''^.g-i v/lr^n students are talking to one 
another in class and distracLi.ng others. 

Criterion 5: Librarian provides effective one-on-ono sessions 
with us^rs. 

Performance standards : 

3. Checks completed term p<'?po): assistance TiJ'^ for sane or 
similar topic. 

2. Propacos "resources" paper in ridvruv* of appointment. 

3. lnc3ud"S all necessary information Tor the resources (title, 
call number, etc.). 

4 . Choos'if; rsources/ tools appropri<^ to user ' s leve 1 of 
expertise and requirements of the paper or project, and m\kos 
adjustments if necessary. 

5. Explains how to use tools with user. 

6. Helps students focus or define tlieir topics as necessary. 

7. Teaches user a transferable search process v;hcn appropriate. 

0. Uses effective communication skills, e.g., checking student 
comprehension through questions, conveying ov/n intorest in topic, 
encouraging followup, etc. 
O 30 
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Bibliographic Instruction 3 

9. Refers user to other libraries, dopartmontc, otc* if 
appropriate • 

Based on Reference Sr Instruction Doi^.'irtment retreat 5/9/87 dcm 
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Reference Desk Service 





WlVCUm OF MtlOM 
llfMM 



(JO I league oemg evaiuaueu 
Date 
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CRD REFERENCE DESK PERFORMANCE STANDARDS/PEER REVIEW FORM 



SELDOM NOT FREQUENTLY 

ENOUGH 



moH srrLE with users 



SOMETIMES MOST OF COMMENTS (also circle 

THE TIME phrases at variance 
with your overall ratlin 



(Verbal and Non-verban 
—VERBAL: 



Uses appropriate tone of voice; 
coMunI Cites well through use of 
clear, concise English; uses good 
telephone etiquette; speaks with 
proper voliae and voice Is not 
too loud or soft; avoids unexplained 
llbraiy Jargon; gears speech to 
user's levtl of understanding 



»»NON- VERBAL r 



Uses appropriate facial expressions 
and gestures; displays cala. esp. 
when working with difficult users; 
shows positive attitude toward users; 
Is approachable, alert, and willing 
to be of asslsUnce; gives complete 
attention to user; respects 
tonfldentlaHlty; sincere; empathetlc 
and establishes good rapport with 
user; puts user at ease; avoids 
talking down to. user 



Allows user to shape question, rather than 
assunlng too quickly what 1$ wanted; takes" 
user to Infomatlon sources; treats all 
users with courtesy, consideration; 
explains Infomatlon; Interacts 
effectively In order to understand 
user's needs; determines appropriate 
level of help needed; suggests users 
return for additional help; refers 
users to subject representatlve(s) 
when necessary 
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..SEinOM. 



TTT 



iv: 



NOT FREQUENTLY 
ENOUGH 



SOMETWES HOST CF COWINTS (also circle 
THE TIME phrases at variance 

with your ov erall ratir 



Consults Kith oth<*r staff aenbers as 
necessary; is sensitive to colleagues' ~ 
Infonution needs and contributes 
constructively; pulls own wight; 
knows when to ask for help; cooperates 
with colleagues; knows appropriate tine 
to assist colleagues working wit h users 



KH0WLED6E OF COUECTIOgg 

Evidences good, general knowledge of ~~ 
CRO collection; has good Judgement for 
appropriate sources for answers; admits 
uncertainty when unsure of proper 
sources; keeps up-to-date on new tools 
in CRD subject areas; shows good 
knowledge of other collections; 
imderstands the card catalog; demonstrates 
knowledge of Geac and OCLC; maintains 
general knowledge of UAL policies/procedures 



V. PtRSOWl qUALiriES/TRAIT S mm i tb AT kimtnti deSk 
(TOWARDS USERS AND COWORKERS) 



Shows good common sense; is flexible and a 

team player; shows sense of humor when 
appropriate; is competent, congenial, courteous, 
dependable, patient, tactful, diplomatic; copes 
well with changing situations; shows responsibility 
as appropriate and can be decisive as necessary; 
goes beyond the minimum required, rather than 
giving up easily 
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CRD REFFJIENCE DESK PERFORMANCE STANDARDS (Evaluation form) 
SaDOM 



NOT FREQUENTLY SOMETIMES MOST OF 
ENOUGH THE TIME 



COMMENTS (exaaplas^) 



I. OOMPtMICATION STYLE 
(Varbal and Non-varbal) 



t. Usas approprlftta tona of voica 



2. OoMMinlcatas vali (usas claar, 
conclsa English) 



3. Usas good talaphona atlquatta 



4. Spaaks vtth propar vol una (voice Is 
nol too loud or soft) 



5. Is varbosa 



6. Usas approprlata facial axprasslons 
(snllas^ looks concarnad» as 
approprlata) 



7. Usas nonverbal comunlcatlon as 
3,pproprlata (head nod» aye contact) 



8. Points vaguely to vhera Information 
can be found 



11. USER INTERACTION AT THE REFERENCE DESK 
\. Allovs patron to shape qitastlon 



2. AssuMS too quickly what user wants 



3. Treats all users with coi^rtesy and 
consideration 



4. Detamlnes what level of help Is 
needed 



5. 
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Displays cairn when working with 
difficult usws 
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^^^^-^^^ CX)»WEKTS (examples...) 
ENOUGH THF Tlir 


6, Displays positive attitude toward users 












7. Explains tools and their relationships 
to users* questions 












8* Friendly toward users 












^* ^ tools vith users (»t iMct 4n 
get ttiM startad) 












10. Intaracts affactivalv with ii«*p in 

• ■ Willi usVr 1 n 

ordar to undarstand his/har naads 












11. Is approachabia (appaars alart and 
willing to ba of asslstanca) 












^ 12. Knows whan to stop giving Infonnatlon 












• «j • 1 1 1 ▼u 1 V iyives oo0ip 1 e T e 
attention to user) 












14% Refers users ^0 cuh i^/^ •>Ani>A«>iMM*.^t. ^ 
• a^vivi 9 W9VI a aiiiijvcT roprosenTot ! vo 

when necessary 












15a Reminds users to return irw AHHt'finnmi 
help 










, 


16a Sensitive to users* needs (respects 
confidentiality! sincere) 












17a Shows concern for user th.rrmgh 
appropriate actions and words; 

III* lib aiiu yoou rOppOrT WiTn 

user; puts user at aasa 












13. Talks down to usar (condescandlng. 
snobbish) 

^ 












19. Usas llbnry Jargon (GEAC, LC, shelf 
list, ate) without explanation 














er|c 



r 



SELDOM NOT FREQUENTLY SOMETIMES MOST OF COMMENTS (•xMpiAS.^) 



20. VoiuntMPs axtra tnfoTMtton when 

approprtatt (axt Whan askad "Where Is 

Miber provide the location and ask 
"Have you ever used the Index to 

Dissertation Ahfi+mrts?*) 












21 • Adalts uncertainty when unsure of 
proper sources 












HI. OOMORKER INTERACTION AT THE REFERENCE DESK 




K Consults irith other staff Mnbers 
when necessary 

u — — — 












*^ 2. Cooperative In ifork with others 
at reference desk 













IV. KN0W.E06E OF COLLECTION 



1. Evidences good, general knowledge of 
CRO collection 




1 








2. Has good Judgement for ap)roprlate 
sources for answers 













V. PERSONAL QU"U,ITIES/TRAITS EXHIBITED AT THE REFERENCE DESK (BOTH IN RELATION ID USr.RS AND COWORKERS) 



t Shows good ccmon setis^. 












2. Possesses t Mnse of humor when 
aoproprlftte 












3. is iXMpetent 












4. Shows self-confidence 












5. Is congenial 












t. is courteous 














' BWJGH -jME TIME 


7> Shors rMDOnstbtl Itv as 
•pproprlat* (dsclslvs.' 










'. ^ 


8. Is flMlbU (oop*s ««ll with 
changing situations) 












9. Shows Initiative (goas bayond tha 












10. Displays patlanca 












11. Is ral labia (dapandabia) 












12. Parsavaras (doas not giva up aaslly) 












y 13« •» ttctful and dlploMtlc 













ADDITIONAL COMMENTS: 
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University of Georgia Libraries 
Athens, Georgia 30602 



November 13, 1986 



Lynn Vestbrook 
Reference Dept. 

2*8460 



Deer Faculty Meaber, 

Ve ere pleated to note that studente in your subject area make good 
use of reference services. With a little help from you» I can learn 
aore about the perceptions these patrons hold of our services. Essen* 
tially^ this help consists of distributing, collecting i id returning 
the enclosed surveys. I have included a brief instruction sheet end a 
self«*addressed envelope. The entire process should take no more than a 
few minutes of class time. 

Tour participation vill improve our understanding of the reference 
process. As a smell return on your investment of time and energy, I 
vill be happy to send you a brief suxomajy ot your clsss* surveys. You 
might: find it useful to knov more about your jtudents* vievs of reference 
servl^i s. 

Thank you for your attention. I hope to hear from you soon. 



Sincerely yovrs. 




Lynn West brook 
Reference Librarian 
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An Equal OpporttuDty / Afftrmahve Actton Institution 



SURVEY DISTRIBUTION INSTRUCTIONS 



1. Choos« any day betwe«n now and December 5, 

2. On that day, distribute the surveys to the students in your class 

3. Uirge them to complete and return th^ surveys as soon as possIbTe 

4. Collect and return the surveys, using the enclosed envelope, by the end of 



«h«t If... 

... I'm not teaching the class listed above? 

Please distribute the surveys to another class in the same department and let 
me know which class eventually received them, 

... I need more surveys? 

Just give me a call at 543-6329. That's my home phone. You can leave a 
oiessage on the machine. Include your name, address and the number of surveys 
you need. I 11 get them out to you right away. 

... I "have questions? 

Glv« me a call at 2-8460, 2-6663, 543-6329. 

... I'd like to fill out a survey myself or know of someone else who wants to? 
Feel free to use extras or call for more. I only ask that you return such 
requested surveys In another envelope so I can keep them separate from the 
others. Every bit of data and every opinion will be helpful so olease f»el 
free to Join In. If your anonymity matters to you, be sure that' the special 
envelope does aot have your return address on it. 
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PLEASE DISCARD THIS FORM IF YOU ARE UNCER 18 YEARS OF AGE 



Thank you for voluntarily participating in this survey. Please do not sign your 
name. After completing this survey, please return it to ycur teacher. Returning 
it indicates your willingness to have the survey used in the research oroject. 

Please answer the following questions based on your experience in the Main Library. 

1. Have you ever wanted to ask for help at the reference desk and decided not to? 
If dO, why? 



2. What do you expect to happen when you ask a reference librarian for help? 



3. Please give a brief description of your last encounter with a reference librarian. 
What did you do? What did the librarian do? 



Are you: female male 

freshman sophomore junior senior 

graduate faculty UGA staff other 

Thank you for vol"ntarily participating. Your supoort is appreciated. 
< IMC rr 




Online Database 
Search Services 




Uuivetsity of Georgia 
1987 ONLINE SEARCHER PERFORMANCE APPRAISAL PROCESS 



I. Choose twelve searches -for each searcher, -following these 
guidel ines: 

Searches cover September - August, distributed -fairly evenly 
Searches cover a variety o-f databases and vendor systems 
Descriptor and -free-text searching are represented 
Searches are long, showing modi -f i cat i on 

II. Examine each search using the -following per-formance criteria: 
Structuring o-f search 

Use o-f term permutations and synonyms 
Use o-f population group terms 
Kr-cwledge o-f commands 
Modi-f ication o-f searches 
Accuracy o-f input 
Overal 1 per-formance 
Make notes on each search. 

III. Rate each searcher by marking a point on the continuum -for each 
cr i ter ion. 



I Unsatisfactory I Satisfactory I Good I Outstanding I 

IV. Discuss the twelve searches and the ratings with the searcher- 
Allow one hour. Make the session instructional, indicating areas o-f 

improvement and alternative search techniques. Modi-fy ratings i-f 
necessary. 

V. Fill out the evaluation -form. 

VI. Submit a copy o-f the -form to the Head o-f Re-ference and the 
searcher . 



ERLC 
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university of Georgia 
1987 ONLINE SEARCHER PERFORf^^CE APPRAISAL 

Name 



Paid searches 


Rr»ady Reference 


searches 


Per-formance criteria 






1. Structuring o-f search 






1 Unsatisfactory 1 Satisfactory I 


Good 


1 Outstandi ng ( 


2. Use of term permutations and 


synonyms 




1 Unsatisfactory I satisfactory I 


iiood 


1 Outstanding i 


3. Use oi population group terms 






1 Un57at isf actory I Satisfactory 1 


6ood 


1 Uutstanding ) 


4. Knowledge of commands 






1 Unsatisfactory I Satisfactory I 


Good 


1 Outstanding I 


5. Modification of searches 






1 Unsatisfactory I Satisfactory I 


Good 


1 Outstanding i 


6. Accuracy of inpul 






I Unsatisfactory 1 Satisfactory 1 


Good 


1 Outstanding I 


7, Overall performance 






1 Unsatisfactory I Satisfactory \ 


Good 


1 Uutstanding i 



Comments 



Other activities 
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1986 ONLINE SEARCHER PERFCRrW4CE APPRAISAL 
Namt J^E DOE 
Paid starchfs 60 
Per-formance criteria 
1. Structuring o-f search 



University of Georgia 
Ready Re-ference searches 10 



I Unsat is-factory I Sat is-factory I 



Good 



I Outstandi ng i 



2. Use of term permutations and synonyms 



I Unsatisfactory I Satisfactory | 



3« Use of population group terms 



Good I Outstanding 



I Unsatisfactory I Satisfactory I 



Good 



I Outstandi ng I 



4. Knowledge of commands 



I Unsatisfactory I Satisfactory I 



5. Modification of searches 



Good 



Outstandi ng I 



I Unsatisfactory I Satisfactory I 



Gooo 



I Outstandi ng I 



6. Accuracy of input 



I Unsatisfactory I Satisfactory I 



Good 



i Outstandi ng I 



7. Overal 1 performance 



I Unsatisfactory I Satisfactory I 



Good 



I Outstandi ng I 



Comments 

Has shown improvement over the past year in all areas. 
Runs well structured searches and modifies them well. 
Needs some improvement in the use of term permutations and synonyms 
and in the use of population terms. 

Other activities 

Attended an ABI/INFORM workshop in February. 

Presented a demonstration of INVESTEXT to other searchers in June. 
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Alfred Taubman Medical Library 

The Unhrcwlty of Michigan • 1 135 E Utherine • Ann Arbor. MI 48109 
Telephone Information* 313*'763-307l 



«iNi INK *^^:^K^•n KVAi.rArrji^i 



In order to cvnliinto nhci Improve our romptitcrlzcd lltcrnturc sonrrhlnp <orvJro^. wo 
would npprccl.itc voiir tnkliM* n tew minutes to rcimplctc thU qtirstlonnnf re. Yon mi\ 
Icnvc the completed questlonnni re either with ^ reference llhrnri.in or in the Llhr.irv' 
sn)if;c<;tlon hex, or drop it in c.nmptK mnl I . 



1. At the time vou requested t»:ls ^enrch, how did voii pinn to ime the re^ult^' 
(check most npproprlnte nnswer) 



RFSKAKCII PR().rK(:r 
' CRANI* PROPOSAL 

PUBLICATION 

lX)(nORAL OISSERTATION 

LEimiRK/TEACHINC 

J_ TERM PAPER 

OTHKR (.specify) 

2. Was the purpose of this search to determine thnt no previous work h.nl heon 
done on this topic? 



YES 
NO 



3. Does this search provide enough relevant citations to meet your needs? 

YES 

but didn't expect to see anything 

NO (please comment) 



A. Were the results of the search useful to you^ 

YES 

NO (please coiwnent) 



5. Among the total citations provided by this search, how many appear relevant 

to the specific question or topic for which you submitted your search request^ 

NONE 

SOME 

MANY 

MOST 

ALL 

6. Among the relevant citations provided by this search, how many are new to you? 

NONE 

SOME 

MANY 

MOST 

ALL 

7. Do you feel that the citations that are both relevant and previously unknown 

to you are worth the cost of the search? 

YES 

NO (please comment) 



PLEASE TURN OVER 
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i« Vas the Clw Itpte betw«.en tubiiitting your search request and receiving your 
results reasonable? 



YES 

NO (please co inent) 



9. Do you feel that the librarl&n who interviewed you understood vour search 
topic? 

YES 

NO (please comment) 



10« Was this your first computerized literature search? 

YES 

NO 

IK How did you become aware of our online search services? 

COLLEAGUE 

REFERENCE LIBRARIAN r 

ONLINE DEMONSTRATION 

LIBRARY BROCHURE 

POSTER 

OTHER (specify) 

12. Which of the following best describes you? 

FACULTY /STAFF 

HOU,$E OFr^CER 

GRADUATE STUDENT 

UNDERGRADUATE 

OTHER (specify) 



13. If you have suggestions as to how any aspect of our sear;.h services can be 
improved, or questions relating either to our search stwices or to the 
search which was just run for you, us« the space below to cotnment. Please 
include your name and telephone number if you would like us to contact you 
regarding your suggestions and/or questions. 
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ALFRED TAUBHAN HEDICAL LIBRARY 
THE UNIVERSITY OF MICHIGAN 

ONLINE SEARCH SERVICE STATISTICS 

Month 

Ytar 

NuAbtr of starch inttrvitNi 

Nuibtr of fttarchti 

NuAbtr run in MEDLINE 

Nuibtr run in NLM non-MEDLINE filti 

Nuibtr run in non-NLM filti 

Avtragt onlint tiit (in linuttf) 

Avtragt formulation tiat (in ainutti) 

Avtragt turnaround tx«t (in Jayi) 

Nuibtr of rtrun itarchtt 

Nunbtr of activt 3DI profilti 
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SEARCHER PROFILE 



SEARCHER: BEGIMNFR 



nilADTCD 


n REQUESTS PROCESSED/ 
I OF TOTAL 


AVG. ONLINE TIME 


AVG. ONLINE TIME - 
MEDLINE ONLY 


AVG. INTERVIEW TIME 


AVG. FORMULATION TIME 


AVG. TURNAROUND TIME 
(IN DAYS) 


n RERUN SEARCHES/ 
I OF TOTAL 


1/1/83 - 
3/31/83 


78/ 
17.8% 


13.8 


12.5 


16 


20 


.83 


2/ 

2.6% 


Vl/83 - 
6/' 30/ 83 


51/ 
13.1% 


9.5 


8.0 


15 


17 


.75 


1/ 
2% 


7/V83 - 
9/30/83 


28/ 
12.8% 


6.6 


6.5 


13 






n 
u 


10/1/83 - 
12/31/83 


52/ 
13.2% 


6.2 


5.9 


11 


10 


.62 


0 


ANNUAL 
AVERAGE 


217/ 
1^% 


9.9 


9.0 


V\ 


16 


.69 


3/ 

1.^% 


DEPARTMENTAL 
AVERAGE 


222/ 
1^.3% 


11.0 


9.^ 


9 


11 

i 


Al 


5/ 

2 . 3/.' 



ALFRED TAUBMAN MEDICAL LIBRARY 
O THE UNIVERSITY OF MICHIGAN 



QUARTERLY SEARCHER PROFILE 



M-83 TO 3-31-83 



SEARCHER 


# REQUESTS PROCESSED/ 
% OF TOTAL 


AVG. ONLINE TIME 


AVG. ONLINE TIME - 
MEDLINE ONLY 


AVG. INTERVIEW 
TIME 


AVG. FORMULATION 
TIME 


AVG. TURNAROUND 
TIME (IN DAYS) 


if RERUN SEARCHES/ 
% OF TOTAL 


A 


^16/ 
10.5% 


13.8 


10.5 


8 


9 


.35 


0 


6 


63/ 


17.8 


12.3 


11 


13 


.56 


2/ 

3.2% 


C 


51/ 
11.7% 


7.7 


6.6 


9 


8 


.22 


0 


D 


52/ 
11.9% 


12.9 


10.9 


8 


Q 


.65 


1/ 

1.9% 


BEGINNER 


78/ 
17.8% 


13.8 


12.5 


16 


20 


.83 


2/ 

2.6% 


E 


15/ 
17.2% 


10.9 


7.9 


10 


18 


.53 


0 


p 

1 


72/ 
16.5% 


12.4 


9.7 


IC 


22 


.36 


3/ 

4.2% 


AVERAGE 


62/ 
1^1.2% 


12.8 


10.7 


11 


15 


.52 


9/ 

1.5% 



ALFRED TAUBMAN MEDICAL LIBRARY 
THE UNIVERSITY OF MICHIGAN 
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Instructional Services 




A PROPOSED METHODOLOGY FOR THE EVALUATION OF 
OF BIBLIOGRAPHIC INSTRUCTION LIBRARIANS: 
BEHAVIORALLY ANCHORED RATING SCALES (BARS) 



DEBORAH A. SOMMER 



UNIVERSITY OF GEORGIA 
MARCH 13, 1986 



The University of Georgia Libraries has a strong and growing program of 
bibliographic instruction. This program consists of orientations to the 
library (both self-guided and formal tourr. - approximately 130 tours each 
year involving 2800 participants), and instruction consisting of lectures on 
basic library skills and subject-related sessions (approximately 220 sessions 
involving 4500 participants). 

As Bibliographic Instruction Coordinator, I am in an unusual situation as 
it pertains to personnel evaluation* As program coordinator, ! receive 
reports from two libraries, Science and Main; I may assign classes to 10 Main 
Reference librarians, approximately 16 other Main Library librarians and refer 
requests to the Science Library. I directly observe and evaluate the 
instructional activities of the Reference Department and maintain records of 
some nature on all librarians participating in the program (approximately 35). 
Because my position is in the Reference Department, my authority to evaluate 
participants outsidt the department is limited. My evaluations of librarians 
outside of the reference department are Indirect (I don't always observe them 
in the classroom) and consist merely of brief written comments to their 
respective supervisors - primarily dealing with their level of activity rather 
than their abilities or effectiveness. My role as evaluator in the Reference 
Department has also been informal. For years, if performance appraisal has 
even occurred related tc B I . , my predecessors and I have informally written 
comments to the Head of Reference to be included on each librarian's annual 
performance evaluation form. These comments have been based upon observation 
of the librarian in the classroom, reports wriiten by t^e librarian, 
instructional materials developed by the librarian, level of participation and 
letters and comments received about the librarian's instructional activities. 



50 

68 



It became evident this year as I prepared my comments that it was very 
difficult to write an essay evaluation of each participant and it was equally 
difficult for the Head of Reference to interpret my comments in concrete and 
quantitative terms. At the same time, my appraisal interviews were more full 
of "praise" (not to mention gratitude) than they were of specific areas for 
development. My role as evaluator is not to determine raises, promotions, 
work assignment, etc., but to provide information to my supervisor, my 
colleagues and myself about performance and to gather information that is 
specifically related to training and staff development. 

Faced with this problem, I decided to explore the possibi 1 i ty of 
developing a performance appraisal methodology/system that could be used 
specifically for appraising the performance of librarians as bibliographic 
instructors and at the same time provide useful information on training neecs. 

Evaluation of reference services is area where there has been 
significant activity in the literature. In a recent article, Powell states: 
"a review of the literature indicated that riuch has been written on the 
evaluation of reference services and reference effecti v**ness" (1984: 89). In 
the same article, he summarizes research on the effectiveness of reference 
staffs. Most evaluation tends to be of twa types - staff availability or 
staff performance. (1984: 94). Only a small portion of the literature 
described by Powell deals with the performance of the reference librarian. 
Those studies that do address this aspect tend to focus more on the length of 
time a patron must wait for service, amount of time spent on various duties, 
and accuracy of reference questions answered. In another recent article on 
the evaluation of reference desk performance, Young has this to say about the 
literature of reference evalijation: 

this literature largely consists of 
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quantitative studies that focus on the 

overall quality of reference services. 

These are of no help in evaluating 

individual performance. (1985:69) 
Young proposes a performance appraisal method, Lehaviorial ly anchored rating 
scales (BARS), that might be applicable to evaluating the performance of 
individual librarians at the Reference Desk (1985:69-75). 

Bibliographic instruction, now an undisputed component of a reference 
librarian's work, is another area where the evaluation literatiTe is growing* 
In a 1980 article, Richard Hume Werking describes B.K evaluation research by 
saying: 

although improvement in library 
use skills (variously defined) is 
the most common object of 
bibliographic education and 
thus evaluation, it is not the 
only one* (80:155) 

In a recent handbook written by the ACRL Bibliographic Instruction Section 
Resea.xh Committee, descriptions are given of other significant types of 
bibliographic instruction evaluation. These include attitude changes of 
library users, impact of BA. on student achievement or library use, the 
effectiveness or comparison of instructional methods, and evaluation for needs 
assessment, program justification or establishment of goals and objectives 
(Werking, 1983:95-102). Though the performance of individual librarians is a 
key component of ary bibliographic instruction program, no significant 
attention has been given to the evaluation of the librarian who does B.I. In 
the same handbook, Adams briefly mentions "measuring teaching effectiveness" 
in her chapter on data gathering instruments: 
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Since few agree on what the characteristics 

of a good teacher are, measuring teacher 

effectiveness is difficult . . . 

about all that can be measured are 

appropriateness of content; usefulness 

of materials or activities; and 

presentation skills. (1983:75) 
No data gathering instruments or formal evaluation techniques were suggested, 
other than asking a colleague to observe or the inclusion of questions about 
the presentation on questionnaires to students (Adams, 1983:75). With only 
Young's suggestion, a look must be made at the literature in the area of 
personnel performance appraisal to determine if BARS would be an appropriate 
performance appraisal technique. 

Performance appraisal has been bantered around in the business, 
psychology and public administration literature for decades (Henderson, 1980; 
Latham and Wexler, 1981; Eiche! and Bender, 1984). This is an area that has 
also received much attention over the years in library literature (Johnson, 
1972; DeProspo, 1971; Berkner, 1979, Reneker, 1982). There ib no doubt a 
consensus on the need for performance appraisal; the methods used for 
appraisals, however, are many and varied. Eichel and Bender take a detailed 
look at currently used performance appraisal techniques (1904). Schneier and 
Beatty identify six objectives of performance appraisal systems: 

1. To provide feedback and improve performance. 

2. To assess training needs. 

3. Identification of promotion potential. 

4. Allocation of organization rewards. 

5. Validation of selection techniques. 
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6. Accuracy in measuring performance. (1979:67-69) 
In my role as appraiser, I am interested primarily in objectives one, two and 
six - to insure that participants maintain an accepted level of performance, 
identify training and development needs and to conduct fair, accurate and 
reliable evaluations. My supervisor, on the other hand, would also benefit 
from an appraisal system that provided information related to the other 
objectives. 

Figure I represents a comparison of four appraisal methods to the 
objectives of a performance appraisal system identified by Schneier and 
Beatty. Of the four techniques listed, behaviorally anchored rating scales 
(BARS), best satisfy all objectives. In Stevenson*s discussion of the 
advantages and disadvantages of a number of appraisal systems used in 
libraries, she presents Figure II which gives the results of six appraisal 
systems when ompared to the following four criteria for a "good" system: 
reliable and valid, job related, standardized, and practical (1984:10). Again, 
the BARS method is the only method to always satisfy at least three of the 
criteria. Stevenson points out that no single appraisal method can satisfy 
each criterion. Therefore, an appraisal method should be chosen based on the 
purpose or objective of the evaluation. 

"Many believe that BARS is one of the most useful aopraisal techniques 
currently available" (Vincellete and Pfister, 1984:105). BARS was first 
introduced in 1963 by Smith and Kendall as behavioral expectation scaling 
(1963:149«155). "A study by Frank J. Landy, et al . , recommends BARS 
especially for the potential benefits they offer for 'counseling and 
feedback*" (Gruenfield, 1981:16). Sttvenson describes BARS as a technique 
that "forces us* tr examine the co^iponents of job performance and to develop 
standards that can be observed for each component" (1984:15). A major 
advantage of BARS "is that they are far more specific in terms of identifying 



FIGURE I 



COMPARISON OF PERFORMANCE APPRAISAL SYSTEMS TO 
PERFORMANCE APPRAISAL OBJECTIVES 



Feedback/ Assessing Identification Reward Selection Measuring 
Development Training of Promotion Allocation System Accuracy 
Needs Potential Validation 



Global 
Trait 



Poor 
Poor 



Poor 
Poor 



Behavior- Very Good Very Good 
based* to 

exc3nent 

Effective- Fair to Fair to 
ness based Good Good 



Poor to Fair 
Poor to Fair 

Very Good 



Poor 



Poor 



Poor to Poor to 
Fair Fair 

Very Good Very Good 
to 

excel lent 



Fair to Good Very Good Fair to 
to Good 
excel lent 



Poor 

Poor to 
Fair 

Good 



Very 
Good 
to 

excel lent 



* Only if behavioral ly-anchorad. 



SOURCE: Schneier and Beatty, 1979:68 
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FIGURE II 



COMPARISONS OF METHODS WITH CRITERIA 
FOR A "GOOD" SYSTEM 



RELIABLE JOB STANDARDIZED 

& VALID RELATED 



PRACTICAL 



ESSAY 

GRAPHIC 
RATING 

FORCED- 
CHOICE 

CRITICAL 
INCIDENT 

M80 

BARS 



YES 



YES 

YES 

YES 
YES 



YES 



YES 

YES 
YES 



NO 
YES 

YES 

NO 

NO 
YES 



YES 
NO 
NO 



NO 



*These items are situational. Managers with good communication skills may 
find the essay format relatively easy to use while those lacking facility with 
words may consider it a difficult task. Some areas of performance lend 
themselves to objective setting while others do not. 



SOURCE: Stevenson, 1984: 10. 
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employee behaviors relative to specific job dimensions" (Schneier and Beatty, 
1979:71). 

The evaluation of specific behaviors has also been used by Kellogg to 
evaluate reference librarians at the University of Arizona (Young, 
1984:72). Adams and Judd describe a method called "goa' analysis" which also 
utilizes a list of behaviors to evaluate reference librarians (1984: 131-145). 
A technique focusing on observable behaviors could be useful and a BARS 
performance appraisal system would meet the objectives of appraisal of 
bibliographic instruction librarians at this institution. 

A variety of methods have been proposed to develop a BARS (Latham and 
Wexley, 1981:48-52; Schwab, Heneman, and DeCottiis, 1975:549-652; and Schneier 
and Beatty, 1979:59-68). To dPvelop a BARS appraisal system to evaluate B.I. 
librarians, the following steps would be taken: 

1. Perform a job analysis. Interview or survey 
supervisors, librarians doing B.I. and others 
knowledgable about the job to identify "critical 
incidents". Have them describe specific incidents 
of effective and ineffective behavior. Compile a 

list or these. For example, some of the incidents might 
be: uses creative methods to present material; makes 
little effort to provide for student interaction; 
presents lecture using a search strategy method- 

2. Develop the job dimensions. Have the same 
individuals categorize these incidents into broad 
overall job categories or "job dimensions". 

For example, Presentation skills. 
Material Development, Participation. 
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I" 

^ 3. Rate the incidents. Have the individuals 

rate on a numerical scale what performance an 
incident reflects. Only those incidents where 
there is a high degree of agreement are retained. 
The numerical value given tj each incident is the 
mean of all the ratings - this is the degree to 
which the incident describes effective behavior 
on the scale. 

To clarify, Figure III shows what a BARS for "Level of Participation" in the 
B.I. program height look like. Some writers include another step. That step 
is to have another group, also familiar with B,L, complete steps 2 and 3. 
The second group's results would then be compared to the first group's. "The 
standard deviation of the ratings for each incident represent the amount of 
agreement among raters regarding the effectiveness level of performance 
described by the incident (the lower the standard deviation, the greater the 
agreement)" (Schwab, Heneman, and DeCotiis, 1975: 551), This process would 
help insure the reliability of the scales. 

The process of developing a BARS appraisal system is crucial to its 
success. Studies have shown that participation in the development of the BARS 
system can lead to more positive feelings toward the appraisal (Silverman and 
Wexley, 1984: 703-711). The effective use of the system also requires the 
rater to systematically observe and document a "representative sample of 
incidents describing the employee's behavior throughout the appraisal period" 
(Latham and Wexley, 1981:54). After the development of the BARS for B.I. 
librarians, it will be necessary for librarians to be observed on a more 
regular basis. The frequency of observation is another element that the 
participating group can decide. Once the system is developed, it is important 
that all raters be instructed in its use. Raters other than the B.!. 

7G 




FIGURE III 
BARS EXAMPLE 

PARTICIPATION IN B.K PROGRAM 

6 Frequently volunteers for library instruction sessions, 

5 Willingly accepts all library instruction requests, 

4 Willingly accepts most library instruction requests, 

3 Occasionally refuses library instruction requests. 

2 Frequently refuses library instruction requests, 

1 Always refuses library instruction requests. 
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Coordinator might include: Head of Main Reference, Assistant Head of Main 
Reference, Head of Government Documents, Supervisors of other Main Library 
B.I. participants, Orientation Librarian, and Head of Science Reference. Some 
studies show that a well-developed BARS can easily be administered by 
untrained students (Harari and Zedeck, 1973:261-265 and Kinicki et al., 
1985:535-549). However, most professionals in the field of per^.onnel advocate 
training to insure the success of an appraisal system. Once the system is 
implemented, it is very important that the system itself be periodically 
evaluated. As job dimensions change, it is necessary to revise the BARS 
measuring them. 

The above process i llustrates why BARS may not always be practical. 
Patten states a major criticism in the use of BARS: "if had one hundred jobs 
and ten dimensions for each job, BARS construction would be a difficult and 
expensive task" (Patten, 1982:150), Constructing a BARS for one activity will 
be time consuming but it should also establish acceptable behaviors for 
librarians to strive for, clearly identify training needs and provide a guage 
of how BA. librarians are performing. A bibl iographic instruction 
librarian's job is generic enough that once a BARS appraisal system has been 
developed, it could serve as a model to be adopted by other librarians in 
similar institutional settings. 
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Pennsylvania State University 

RECOMMENDATIONS FOR PEER REVIEW 
OF CREDIT INSTRUCTION 

Curricular and Instruction Affairs Connittee 
Library Faculty Organisation 



1. Sunnative evaluation should be under the direction of the appropriate 
department chief or campus academic officer, who will ensure that a 
longitudinal representation of peer evaluation of credit instruction 
is included in promotion and tenure dossiers. 

2. Suiftnative evaluation will be performed by tenured librarians of the 
Pennsylvania State University who have taught a course for credit. 
Selection of the evaluator and date(s) for evaluation will be made by 
mutual agreement between the librarian and the appropriate department 
chief or administrative officer. 

3. Sumnative evaluation will be carried out at least once per course/ 
semester. 

4. Individual librai ians are encouraged to arrange for formative evalua- 
tions of their leaching. The Coordinator of Instruction will assist 
librarians in this process. 

5. An evaluative instrument should be u^zd vhich would codify responses to 
individual teaching performance in a consistent manner. 

6. Evaluators are encouraged to have pre* and post* evaluation discussions 
with the librarians being evaluated. 

7. Training of evaluators will be arranged by the Coordinator of Instruc* 
tion. 

8. Peer evaluations will be placed in the librarian's conf identii.1 file. 



C. Whittington 
April 20, 1987 



ERIC 



70 

61 



iHh FhNNSYLVANlA STATE UNIVERSITY 

The University Libraries 
Peer Review of Credit Instruction 



Instructor: 
Course: 



Evaluator: 



Semester: 



Class Evaluation ; 

Al. Rate the clarity and organization of the 

instructor's presentation. 
A2. Rate the instructor's demonstrated knowledge 

of the subject of the class. 
A3. Rate the instructor's demonstrated enthusiasm 

foi^ the subject matter. 
AA. Rate the instructor's effective use of class 

time. 

A5. Rate the instructor's use of examples and 

illustrations. 
A6. Rate the instructor's preparation for class. 
A7. Rate the instructor's response to student 

questions and conments. 
A8. Rate the instructor's creation ol an 

atmosphere conducive to learning. 



Date: 



lowest 



ra 



ing 



2 

2 

2 

2 

2 

2 
2 



3 

3 

3 

3 

3 

3 
3 



average 
rating 



4 
4 



5 

5 

5 

5 

5 

5 
5 



highest 
:ing 



ral 



6 
6 
6 
6 

6 

6 
6 



Convnents: 



Course Materials Evaluation : 

Bl. R/ite the clarity of course objectives. 12 3 4 5 6 7 

B2. Kate the logic of the presentation of 1 2 3 A 5 6 7 
the material. 

B3. Rate the relevancy of handouts, assign- 12 3 A 5 6 7 

ments, and tests to course objectives. 

B4. Rate the clarity of criteria for grades. 12 3 4 5 6 7 

Comments: , 



Overall Evaluation. 
Comments: 



1 2 3 4 5 6 7 



Indiana University ^J^JL^^^' 

It 

Librarian 



To; V131 kt^m Front Jull» Bobay^ Instruction 



D»pt. t Engllflh D»pt. 

Subji Library Toura Dat»j March 29, 1986 



W» ar» d»v»loping a n»v pr^^a»ntation on libriry r»a»arch and 
using th» lU Libraries. This nsv prsasntation vill rspiacs th» 
tour/prsasntation currently offarad to W131 claaaaa. Wa plan to 
hava tha nav prograa raady thia fall. 

To anaura that our nav program a«^ata vh«r naada of Baaic 
Coapoaition atudanta, va vant to find out hov you faal about tha 
library inatruction givan to your claaaaa in tha paat. Wa alao 
v^nt to knov vhat library akilla you faal ari» aoat important for 
your atudanta aa thay vork on W131 aaaignaanta. 

Plaaaa taka a fav ainutaa to raapond to tha quaationa balov. 
Your cooparation vill giva ua valuable information that va can 
uaa aa va plan tha nrv program. 

Plaaaa uaa tha ancloaad addraaaad anvalopa to raturn thia 
quaationnaira to ma by April 4, 1986. Thank you. 



1. Hov oftan do you uaa library matariala for your ovn raaaarch? 

Mora than cnca a vaak 

At laaat onca a month 



Laaa than onca a month 



2. Do you uaa aubjact indaxaa othar than tha H^A gibiiSOSBCbZ 
for your raaaarch? YES NO 

COMMENTS 



3« For hovr many aamaatara hava you brought W131 claaaaa to 
tha library for a praaantation? 

4. Nama tha moat racant aamaatar that you'va achadulad a library 
praaantation for your W131 claaa. 



Ox 



ERLC " 

hnimiimrfTiaaaa 



S* Your V131 class (••> vhlch cam for m library prsssntatlon 
during thm momt r#c#nt •••••tsr r#c#iv»d at 



_alld» prasantatlon In tha Llbrarlas' claaaroom 

._guidad tour of tha library vith Indaxaa and 
catalogs axplalnad vlthln th ^ valklng tour 

.a conblnatlon of tha slldas and a valklng tour 

Othar (Cascrlba) 



6* In gan«rC(l^ hov satlsflad wara you vlth tha library session: 
Vary satlsflad 

Satlsflad^ but could ba Inprovad 

Dlssatlsflad 

COMMEHTS _ 



7« If you r^calvad tha folloving prasantatlons In your K,ost 
racant library sasslon^ plaasa Indlcata your asaassmant ol that 
alaaant t 

Racalvad right Should ba 

anount of tlma 5 4 3 2 1 axcludad 



Prasantatlon Prasantation 
was affaotiva 5 4 3 2 1 vas inaffactlva 

COMMENTS 



Racalvad right Should bi 

amount of tlma 5 4 3 2 1 axcludad 



Prasantatlon Prasantatlon 
vas affactivit 5 4 3 2 1 vaa Inaffactlva 

COMMEHTS 



c. Ltiffio fifi Bt£lfiiSieai iDdtxti 



R*c*lv»d right 
•Mount of tlM* 

Pr*«*ntatlo;:« 
vaa •ff»ctlv» 



COHHENTS 



Should 
•xcludtfd 

PrvMntatioA 
was ln»ff»ctivi 



R»o»lv»d right 
•nount of tin* 5 

Pr»«»ntatlon 

VM •ffvctlv* S 



3 
3 



2 
2 



Should b* 
•xcludvd 

PrvMntatlon 
was ln»ff»ctlv» 



COHHENTS 



a. Did your library ■•■■ion owit son* Infornatlon that you 
thought was Inportant? 

COHHENTS 



9* If it «»r» avallabl»» would you davota « aacond claaa aaaalon 
In your W131 class to a nora In-dapth aaaslon on how to uea tha 

library? YES NO 

COHHENTS 



10. Tha raaearch papars turnvd in by ay studanta ahovad avldanca 
that thayt 

V»ry Hot 

Succaaaful S 4 3 2 i Succaaaful 

wtct iUSCtlllui is ifiSlilDfl IBBCSBCiiit Btclgdlsil iC&iSitS iQ 

V»ry Not 

Succaaaful 5 4 3 2 1 Succaaaful 
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Indiana tJ?iiversity 
W131 LlBkARf PkOGRAM EVALUATION 

This year is the first for the W131 library presentations. We 
would like to know whether you found the presentations helpful. 
Please help us by answering the questions below. We will use your 
responses to inprove the presentation. 

FOR EACH QUESTION, PLEASE CIRCLE THE ANSWER THAT BilST REPRESENTS 
YOUR OPINION. 

1. I went to the Main Lr.brary and completed the self -guided 
tour: 

a) before the library presentation 

b) after the library presentation 

c) I chose not complete the tour 

d) I was not given the self -guided tour by my instructor. 

2. In doing library research, I was able to find books on my topic 
in the card catalog.?: 

a) yes 
b> no 

c) 1 did not need books tor my research. 

3. In doing library research, 1 was able to find articles on my 
topic in a periodical index: 

a) yes 

b) no 

c) 1 did not use periodical articles. 

4. 1 was able to find the periodicals 1 needed using the Selected 
List ot Peri odical Holdings ( red book ) . 

a) always- 

b) most of the time 

c) sometimes 

d ) never . 

b. The library presentation (Circle as many answers as apply to 
you ) 

a) was so boring I almost went to sleep. 

b) was interesting. 1 did not know there was that much good 
information in the library. 

c) gave me more information than I wanted to kuow. 

d) was too fast and confusing. 

e) ^;old me a lot of stuff 1 already knew. 

f) other (please explain) 




6. The ^loat umIuI part of th«i library praaantation waa' 

a) the self -guided tour 

b) the information about the card catalog and subject terms 

c) the section covering indexes to magazines and scholarly 
Journals 

d) the section about newspaper indexes. 

7. The le^s t useful part of the library presentation was: 

a) the self -guided tovr 

b) the infomation about the card catalog and subject terms 

c) the section covering indexes to magazines and scholarly 
journals 

c) the section about newspaper indexes. 

8. I asked tor help at the reference desk 

a) yes 

b) no 

c) don't remember. 

9. All W131 classes should have this class presentation 

a) strongly agree 

b) somewhat agree 

c) don't know 

d) somewhat disagree 

e) strongly disagree 

10. Comments' 
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Indiana University 

Toi M131 AI'» From: Julxe Bobay, Instruction 

Librar^xan 

Dept.: English Dept. Dept : Library E17i2 

SubJ: Library Presentations Date: November 5, 1986 



fifter trying out the W131 library program for one semester, 
we'd like tc ask your help in evaluating its effectiveness. 
We will use your responses to this questionnaire to make 
modifications and improvements in the present at iovu 

Please use the enclosed envelope to return this questionnaires to 
me by November £1, 1966. Thank you. 

************************************************************* 

1. For how many semesters have you brought W131 classes to 
the library for a presentation? 



£. Did you hand out the W131 Self-guided tour before the library 
present at ion? 

YES NO 

a. If yes, did you check that your students did it? 

YES NO 

Please describe your method of checking: 



What percentage of your students do you think took the 
sel f-gui ded tour? 



b. Do you think the self-guided tour xs an effective way to 

orient students to the Main Library? 

Effective 5 4 3 2 1 Ineffective 

WHY OR WHY NOT? 
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3. Plft 



ntfi of th« library presentation: 



Received right Should be 

amount of time 5 A 3 2 1 excluded 

Present at i on Present at i on 

was effective 5 A 3 2 1 was ineffective 

Comments: 



^- E^CiSdi^^i iD^exes < inc 1 ud ing in-class exerc ise) 

Received right Should be 

amount of time 5 4 3 2 1 excluded 

Present at i on Present at i on 

was effective 5 4 3 S 1 was ineffective 

Co*flment s : 



EiD^lDS § egni2^l£§i < including in-class exercise) 

Received right Should be 

amount of time 5 A 3 1 exclud€?d 

Presentation Presentation 

was effective 5 4 3 2 1 was ineffective 

Comments: 



^« ^ewsQager l!2^^>i€§ 

Received right Should be 

amount of time 5 4 3 2 1 excluded 

Present at ion Present at ion 

was effective 5 4 3 2 1 was ineffective 

Comments: 
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5. Th* research papers turned in by rny students showed evidence 
that they: 



*• were successful in f inVdlQa aEB!:9B!21§te books in the library 

Very Not 

Successful 5 4 3 2 1 Successful 

t>« we|2e sueee'^lfl^i in iQcating aBE!:2Bi:i^te B^riodical and 

D^wsEaeer §rt i,cl.es 

Very Not 

Successful 5 4 3 2 1 Successful 

Comments: 



6. a. Overall, how satisfied were you with the library se«>sionj 
Satisfied 5 4 3 2 1 Dissatisfied 



b. Do you plan to schedule library presentations in future 
semesters? 

YES NO 

COMMENTS 



7. Would you be interested in a slide/tape version of this 
presentation to be administered by you rather than a librarian"^ 
Why or why not? 



-OVER- 




a. For firftt-y««r W131 APsi 

Did you fmml thm library s»gm«nt of your .orientation session 
was useful? Do you have suggestions for improving the 
presentation Me give to new PI*s about the library presentation? 



ADDITIONPL COMMENTS: 
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Indiana University 



gyaLUftllON OF ELEMENTQRY COMPQSIIION LIBRGRY PRESENIGIION 

During the Fall 1986 semester, the Libraries offered a new 
presentation to 70 sections of W13i. The lesson consisted of a 
self-guided tour to e administered by the ftss.ociate Instructors 
and an in-class slide presentation, narrated by a librarian, 
which covered basic library skills. The classroom session 
include ^ ungraded exercises where students used sample indexes to 
find articles on ^ topic. 

QUESTIONNPIRES: 

56% Return Rate for Associate Instructors (31/55) 

(questionnaire rriv^iled to each participating ftl) 

SB% Return Rate for Students (685/1309) 

(questionnaires mailed to each ftl to administer in class at 
the end of the semester) 



CONCLUSIONS; 

It seems that our users appreciate the fact that this lesson 
makes a very complicated process less complicated, and goes a long 
way toward achieving our goal: "W131 students will be able to 
identify and find in the lU Libraries sufficient resource 
materials to complete a shore research paper. " Even students who 
complained that the session was boring and told them a lot of 
stuff they alreacy knew recomrrended that all W131 classes should 
Q^t the lesson. (Perhaps that is similar to the benefits of 
taking foul-tasting medicine.) 

There was fairly broad consensus that the self-guided tour was a 
good idea, but lacked any means of motivating the students to 
actually do it and J'-arn anything from it. The most highly rated 
portion of the lessor* by both students and ftl's was the section 
on periodical indexes and finding journals in the library. Many 
commented that we needed to stress this area even more. The card 
catalog section was generally less well-received, but seen as 
valuable by a significant number of students and ftl's. The 
section on newspaper indexes was very poorly rated by students 
and AI's. 



B^SeiOEOeDd £ t i o n s 

The progress weWe made unus far in teaching basic Itbrury skills 

to elementary composition classes is encouraging, but areas 

remain which need improvement. We recommend eliminating the self-guided 

tour because it is not effective. Other ':^roblems (boring, teaching 

students what they think they already know, covering material 

which may or may not be valued by an indiyxdual Associate 

Instructor, the lack of realism in the simulated exercise) can 

at least theoretically be answered by designing a computer— 
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Msittt«d iMSon to r«plac« thm slid« show. This Mould •nabl* us 
to put ths burdsn of iMrning back on thm studsnt, and would 
•ncour«B« ^hmm to take an activs part in thm process. It would 
also allow therti to choose which skills they need to learn. 

Feedback to the instructor could be built in through the use of 
an online quir, and hands-on experience could be provided through 
a follow-up library exercise. 

While these potential benefits of Computer-Assisted Instruction 
have been apparent to us for some time, ^m never felt the 
university had a system which could support 4,000-6,000 students 
anc* be simple enough to use and administer. The proposed Student 
Information Network seems to have the potential to provide those 
capabilities, and so we recommend designing a test Cfil lesson for 
implementation next fall. 



Indiana University 

HIGHLIGHTS FROM W131 QUESTIONNOI Rr S 

Selfzayi^ed tour 

—Students - did not take 

37% - least useful part 

-«I's - 37% handed it out 

Problems cited: getting students to do it. Sorne question its 
effectiveness even when it is done. 

Q^Cd Catalog 

—Students - 33% least useful part 
£0% most useful part 

65% able to find booke 
30% did not need books 

—Pis —3.6 (right amount of time index) 
3.8 (effectiveness index) 

Comments: Typical comment: "This was covered fairly well, but I 
think that students know this better than most things in the 
library; and, in a crunch, some of this time could be 
sacrificed. " 



3. {Igcied ical indexes 

—Students: &S% able to find articles in index 

S0% most useful part of presentation 

— ftl's: 4.1 (right amount of time index) 
3. 9 (effect i veness i ndex ) 

Comments: stress scholarly indexes more. More emphoDis should 
be placed on this. Exercise good idea; sometimes r ashed. 



^- Einding a Periodical 

-Students: 63^^ (always or most of the time able to use the Red 

Book to find periodicals. ) 
Sl7% (sometimes able to use the Red Book) 

-Pi's: 3.8 (right amount of time index) 
3.9 (effectiveness index) 
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Commentst HMdm mor# t ifn# and attention. Mor» thorough 
cov»r«g«. Common th«m»s "I lik« th» ^xmrcinm *- I wi%h it had 
stuck with my •tudvnts b«tt»r. 



5- NtiSSBftBtC iodSHCS 

-Students: 12^ most useful part 
18% iMst useful part 

-ftl's: 3.4 (riyht amount of time index) 
3.! (effectiveness index) 

Comments: hurried, rushed, weak. Not enough time. Disagreement 
on usefulness of newspapers for college level work. 



®eD€C^i iCQECISssions 

-Students: 3&% - was interesting 

35% - told me a lot of stuff I already knew 
£2^ - so boring I almost went to sleep 
18^ - gave me more information than I wanted 
11% - wa? too f a »t and confusing 

"Oil wl31 classes should have this presentation:" 

B2"X of the students strongly or somewhat agree 



-ftl's: 

Were students successful? books: 3.7 out of 5 

articles: 3.4 out of 5 

Were yoii satisfied with this presentation? 3.9 out of 5 

Will you do this again? 9&% yes. 

Comments: many asking for more hands-on experience; "learn by 
doing, " "gap between what they see in one room and 
what they will actually do." 
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Indiana University 

Evaluation of Seminars on 
Library Support for Instruction 

The purpose of this seminar was to provide information about services 
that the I.U. Libraries provide to assist Instructors and their students. 
Please help us to evaluate the session by completing this form. 

(1) How useful was the session? 

very useful ^neither useful nor useless 

^useful generally useless 



(2) Was the session 

too long, too short, or ^about right? 

too fast, too slow, or ^about right? 

too elementary, too advanced, or ^about right? 

(3) How interesting was the session? 

very Inteirestlng ^neither dull nor Interesting 

Interesting dull 

(4) How could the material have been presented more effectively? 



(5) What kinds of services would you Mice the Libraries to offer that were 
not discussed today? 



(6) Are there specific topics on bibliographic access, research resources, 
or others that you would like to see presented In future sessions? 
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Commvntss n«»d» more time and attention. More thorough 
coverage. Common themes "I like the exercise - I wish it had 
stu«rk Nith my students better. ** 



5* Nft^f iQ^&>l^& 

-Students! 1^ most useful part 
16% least useful part 

-ftl' ss 3. 4 (right amount of i.xme index) 
3.1 (effectiveness index) 

Comments! hurried, runhed, weak. Not enough time. Disagreement 
on usefulness of newspapers for college level work. 



-Students! 3&% - was interest inq 

35% " told me a lot of stuff 1 already knew 
B£% - so boring I almost went to sleep 
iB% - gave me more inform^^tion than I wanted 
- was too fast and confusing 

"All wl31 classes should have this presentations" 

6Z% of the students strong'* y or somewhat agree 



-AI's: 

Were students successful? booUss 3.7 out of 5 

articles: 3.4 Ov.t of 5 

Were you satisfied with this presentation? 3.9 out of 5 

Will you do this again? 3S% yes. 

Comments! many asking for more hands-c experience; **learn by 
doing, " "gap between what thev see in one room and 
what they will actually do." 
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FACULTY SEMINRR EVRl.UPTIONG 
WEDNESDAY and THURSDAY, FEBRUftRY 26 and 37 y 1366 

1. Did the session cover what you enpected? 
18 — Yes 0 — No 

2. Which parts of the session did you ^ind particular/ valuable? 

17 — Description of available services and databases 
8 — Outline of advantages/disadvantages 

12 — Outline of costs 

12 — Outline of search techniques 

10 — Demonstrations 

Comments : 
I especially liked seeing it work. 

Encellent presentation — very informational — I enjoyed it 
very much. Thank vou. 
All were useful. Perhaps more information could be given on how 

to type in the phrases — arrangement of phrases, use of 

parenthesis, etc. 
Need to follow up to get information on how to download to disk. 

3. Which parts of the session were less valuable? 

0 — Description of available services and databases 

6 — Uut 1 ine of advantages/d i sad vantages 

3 — Outline of costs 

O — Outline of search techniques 

2 — Demonstrations 

Comments : 

OK though. Didn't spend much time on it 

(advantages/disadvantages) and that wa*3 fine. 
Less val lable only because I was familiar with advantages 

and d isadvant ages. 
None. 

Pfter getting the other information, I c'scided I really 
^9Dlt want to search by myself • 

Was the session (1.5 hours): 

0 — too long*^ 
0 — too short? 

18 — about r I ght ? 

CJ^mment s : 

H^bout right at introductory level. Good sesion! I would 
like more specific search t»^aining, but given the t \me 
limitation of this introductory session, it was well done. 
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5. Would you attend a more detailed seminar in your subject 

area if it were offered? ^ 

1 — No 

14 — Yes (please indicate area: 

Bus i ness/Soc i a 1 Sci ences — 1 
Communication — 1 
Educat ion — I 
Education (Higher) — 1 
Education/Music — 1 
English Literature — 1 
Geo logy /Geophysics — 1 
Humanities, etc — 1 (am a librarian) 
Information Specialist — 1 (any area. I would 
like information on search strategies in more 
detail than this session offered • ) 
Instruct ional Technology — 1 
Journalism — 1 
Law — 1 

Library & Information Science — 1 
Linguistics — 1 
Medical Sciences — 1 

3 — Undecided 

After today's session, are you planning to investigate 
further any of the services we discussed? 

0 — No If yes, which ones? 

16 — Yes 

2 — Undecided 5 — BRS 

12 — BRS Pfter Dark 
6 — DIALOG 

11 — DIALOG* s Knowledge Index 

7. Do you thini' the University Libraries should consider offering 
some type of formal /informal consulting services for people on 
campus who are personally using these systems? 

0 — No 
13 — Yes 
5 — Undecided 

9, Should the University Libraries offer forroal training in online 
bib A iograph ic search ing for people such as yourselves'^ 

12 " Yes 
0 — No 
5 — Undecided 

Comment s : 

Believe it is available — "Computer Literacy" II 
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#• ijjA^ UtiiX^C^ ttiy»,iO«5mCalifornia - Santa Barbara 

Purvey: 14406 TEXm TAUGHT INT 1 0100 INT 

1. Would you recommend this class to your friends as an elective? 



(a) yes, no reservations 

(b) yes, but I'd have a few reservations 

(c) yes, but I*d have serious reservations 

(d) no, I wouldn't recommend it 

2. Knowing what the course covers, if you had not already taken it, would you 

(a) definitely want to take it, 

(b) probably want to take it, 

(c) be undecided about taking it, 

(d) probably not want to take it* 

(e) definitely not want to take it. 

3. How well does your instructor seem to know the subject matter of the 
course/* 

(.y) truly exceptional knowledge 

(b) thorough knowledge 

(c) adequate knowledge 

(d) lacks important knowledgr 

(e) inadequate knowledge 

4. Exclusive of time spent in class or labs, how many hours did you spend on 
this class? 

(a) 0-2 (b) 3-4 (c) 5-6 (d) 7-8 (e) 9 or more 

5. This course helped me develop specific skills, competencies, and points 
view needed to use the UCSB Library. 

strongly agree agree undecided disagree stronglv disagree 
(a) (b) (c) (d) (e) 

6. How much do you feel you have learned in this class? 

(a) a great deal 

(b) quite a lot 

(c) some 

(d) very little 

(e) almost nothing 

7. The instructor's explanation of difficult material is 

(a) generally very helpful, 

(b) helpful. 

(c) not usually helpful. 

(d) almost nonexistent. 

8. Directions for course assignments are clear and specific. 

(a) never (b) seldom (c) usually (d) almost alvays 
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say - 



9. For me, the assignments are 

(a) too long. (b) about the right length. (c) too short. 

10. My overall evaluation is that the t .xt vas: 

Of Little value Satisfactory Excellent 

(a) (b) (c) (d) (e) 

11. In my opinion, the assignments are: 

too easy of reasonable difficulty too difficult 

(a) (b) (c) 

12. The assigned text for this course vas: 

(a) too superficial and redundant with lecture material 

(b) interesting as a supplement to lecture material 

(c) interesting and comprehensive all by itself 

(d) dull and difficult to read 

13. The assignment exercises aided my learning to use the library. 

strongly agree agree neutral disagree strongly disagree 
ta) (b) (c) (d) (e) 

14. The exam is reasonable in length and difficulty. 

strongly agree agree uncertain disagree strongly disagree 
<a) , (b) (c) (d) (e) 

15. The grading procedures are clearly explained. 

stronglv agree agree uncertain disagree strongly disagree 
<a) (b) (c) (d) (e) 

16. Overall, how w \d you rate the value of the review session? 

(a) Excellent Good (c) Satisfactory (d) poor (e) Very Poor 

17. The amount of material cov by the course was: 

Too Little About Right Too Much 

<«> (b) (c) (d) (e) 
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18. When in the quarter did you work on you exercises? 

(a) every week 

(b) the last week 

(c) the last two weeks 

(d) the night before due 

(e) at the beginning of the quarter 



19. Would you have liked to learn the information another way? 

(a) lectures in class 

(b) video lectures 

(c) computer instruction 

(d) No 

-f 

20. Have you any comments to make about the text, exercises, the schedule, the 
library, the review session, or assistance at service points? j 
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Selected Reading List 






Office ef Manag«ment Studies. Association of Research Libranes 
1527 New Hampshire Ave^'jo. N.W., Washington, D.C. 20036 • 202-232-8656 
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USES OF SPEC KITS 



The Systems ond Procedures Exchar^ Center (SPEC) is o 
cleannghouse operoted by the Associotion of Research 
Ubrories. Office of Monagement Studies thot provides o centre! 
source of timely information ond moterials on the monagement 
ond operotions of targe ocodemic ond research hbrones. It 
focilltotes the exchange of knowledge ond documents thrajgh 
SPEC Kits, which ore distributed ten times each year to APL 
members ond other Interested librories The Kits include 
tcpicaily-arronged groupings of unedited primory source 
document?. - selected for their volue to odministrotors ord 
decision-makers - that illustrote o wide rnnge of olternati^e op- 
pfoaches to specfc issues. 

Kit documents come from generol membership surveys ond 
from selected librories contocted directly by SPEC, ord most 
Kits ore produced within six months of surveys The documents' 
volue comes from their voriety of ideas, methods, ond solutions. 
They are not viewed os fir^shed products, but rother os points of 
departure for o library's planning efforts ond os stimutants to in- 
novative opproaches to problem-solving. As such. Kits do not 
present onswers or prescriptions for ony one librory; instead 
they Illustrate how selected ARL members ore plonning for or 
dealing v^nth particutar issues. The worth of ony one Kit to o par- 
ticulor librory will depend upon ttie specific topic covered ond 
the librory's stoge of development in that orec. 



Moteriols ore selected occording to the ollowing cnler'n 

• Presents on approach of potentiol volue to odministrotors 
ord decisiorvmokers 

• Timely, ord deoling directly with the topic under con- 
sidorotion 

«♦ Probability of opplicotion cf deos cr thinking to other 
librory situotions 

• lllustrotive of octuol proctice rother thon theoreticol 

• Understondable. *'eadable communicotion 

All together, the moteriols should provkje o ronge o^ olternotive 
opproaches that complement each other, provide vonety. ord 
stimulate comparison ond controst 

Librories can toke odvontoge of tt^ Kit compilotions in o 
number of woys. Administrotors can evoluote the ossumptions. 
methods, ond results of other librories' opproaches: compare 
ond controst them ord use the learnings in their own situotions. 
Librory staff members con use the kits os professional develop- 
ment ord current oworeness tods. Committees ord tosk forces 
con use them to begin o review of current proctices And the 
Kits con identify other persons or ploces to contoct for further 
informotion. Back-up files in the SPEC office olso ore ovoiloble 
for loan to member libraries. In odditioa SPEC will conduct on- 
demond surveys cr onalyses geored sp€ -^ificolly for o single 
librory. 



CVMLUMII^IN 

Kit Title /Number 

1 Which uses did the librory make of this Kit? 



2 Please indicote how useful the Kit wos for these purposes. 

Very Useful D Quite Useful C Somewhat Useful □ Not Useful 

3. Do you have suggestions for this Kit or for future Kits'^ 



(optional) NAME _ 

LIBRARY 
PHONE. 



Please return this form to the SPEC Center. OMS/ARL. 1527 New Hompshire Ave. NW. Washington, DC 20036 

1/82 



ERLC 



